A vendor pitch for an AI ad tool usually lands on one sentence: this will tell you which ad is going to win before you spend a dollar on it. It sounds like exactly what a business owner wants. It's also the sentence that should make you slow down, because predicting how an ad will sell before it runs is a fundamentally different problem than describing what's on the screen, and most of the tools making that promise are not built to solve it.
What prediction actually requires.
Picture a restaurant owner choosing between two Facebook ads for a weekend special. One tool tells her, with a confident percentage, which ad will bring in more bookings. That's not a reading of the image. That's a claim about cause and effect in the real world: this creative, shown to this audience, will produce that many more sales than the other one would have produced on its own.
Figuring out which marketing dollar actually caused a sale, as opposed to a sale that would've happened anyway, is so hard that big companies hire specialized statisticians who spend months building and testing models just to answer it. And those are companies with millions of dollars to spend on getting it right. A recent piece in AdExchanger makes the case plainly: most "AI-powered" ad tools run on the same kind of technology as ChatGPT, software trained to guess what word comes next in a sentence. That's genuinely useful for writing text. It's not the same as figuring out what actually made someone buy something. The piece argues that many vendors use the phrase "AI-powered" as marketing language that obscures what's actually happening under the hood, and that this repackages an old problem: a measurement model nobody can actually inspect or validate, now with a shinier label.
The stakes aren't abstract. The same piece points to Alaska Airlines, which moved to an open-source measurement model with a budget in the nine figures, where a small forecasting error doesn't mean a wasted afternoon. It means millions of dollars sent to the wrong channel. A small business owner isn't risking that much money, but the mechanism is the same: a wrong confident number moves your budget somewhere it shouldn't go, and you have no way to catch it because the tool never told you it was guessing. What you need before you spend is a way to catch that guess, and it's the problem we're building Knowing to solve.
The tool that's right most of the time is the one you stop watching.
Here's the part that doesn't get said enough: a prediction tool that's wrong sometimes is actually less dangerous than one that's right almost all the time. That sounds backwards until you think about how people actually behave around tools they trust.
A data scientist named Cassie Kozyrkov describes this perfectly using two hypothetical coworkers, in a widely shared essay called the AI reliability paradox. One, "Chris Careless," succeeds 70% of the time. Because people see him fail regularly, they build safety nets around his work: second opinions, checks, a habit of double-checking before it matters. The other, "Ronnie Reliable," succeeds 99.99% of the time. Because people almost never see him fail, they stop checking. They hand him more responsibility and less oversight, right up until the one time he's wrong, at a scale nobody built a safety net for.
By failing to understand that there's a crucial difference between good and perfect, leaders can turn the blessing of a good worker into the curse of the high performer.
Kozyrkov's point is that human attention doesn't scale down proportionally with a system's failure rate. It rounds down to zero. A tool that's right 9,999 times out of 10,000 doesn't register to most people as a tool with a failure rate at all; it registers as a tool that doesn't fail. That's the exact moment oversight disappears, which is also the exact moment it's needed most, because at scale, rare failures stop being rare. As she puts it, quoting a line from site reliability engineering: "When you increase the scale, you'll meet the long tail." In plain terms: when you run enough ads through a tool that's wrong once per thousand times, you will eventually hit that one wrong prediction. And it won't feel like a one-in-a-thousand moment. It'll feel like it cost you the ad budget for the month.
Here's the trap underneath all of it: when a tool shows you a confident number, like an 87% chance this ad wins, you trust it more than you should, simply because it has math attached. A confident guess with a percentage on it feels like evidence. It isn't automatically evidence of anything. A commenter on the LinkedIn thread discussing the piece connected this directly to Nassim Taleb's idea of the black swan: a failure that looks impossible right up until it happens, precisely because a long run of success trained everyone to stop expecting one.
What a tool that knows its limits actually looks like.
So what can an AI ad tool actually do? Here's the honest version. None of this means AI is useless in advertising, or that every claim about it is hollow. The AdExchanger piece is specific about where AI legitimately helps in measurement work right now: spotting patterns in a pile of data, explaining what a model actually found in plain words, catching an anomaly in your numbers that would take a person hours to find. What it can't yet do is replace the causal modeling itself, the actual work of figuring out which investments caused incremental revenue and which just captured a sale that was going to happen regardless.
That line maps onto the difference between evaluating a creative and predicting its sales. A tool can read an ad honestly: what's actually in the image, whether the words match what you've said your brand stands for, whether a viewer has anywhere to land before scrolling past, whether the offer is specific enough to be believed. Those are checkable things, grounded in research on how attention and memory actually work, not forecasts of a future nobody can see yet. A tool that tells you your ad is unfocused or your promise is vague is making a claim you can verify by looking at the ad. A tool that tells you this ad will outsell that one by 23% is making a claim nobody can verify until the campaign is already over, and by then the budget is already spent.
Not "how accurate is your AI," since every vendor will give you a number that sounds impressive. Ask instead: what did your predictions miss last quarter, and how did you find out? A team doing real measurement work has an answer, because the AdExchanger piece lays out what that discipline looks like in practice: a dedicated budget for running controlled experiments, checking forecasts against what actually happened, and watching for a model that keeps missing in the same direction. Most vendors won't have a good answer to this, and that's itself useful information. A vendor with no answer isn't measuring anything. They're guessing with better formatting.
The honest version of AI in advertising sounds less impressive on a sales call. It doesn't promise to pick the winner before you spend the money. It tells you, in plain terms, whether the thing you're about to publish is built the way the evidence says a working ad is built, and it says so without pretending to know what it can't know. That's a smaller promise. It's also the one that's actually true, which turns out to matter more than it sounds like it should, right up until the quarter a confident wrong number costs you the budget nobody was watching.
Frequently asked
Common questions.
Can AI actually predict whether an ad will perform well before it runs?
Not reliably. Predicting sales requires untangling cause and effect, the hardest problem in marketing measurement. Large language models are built to generate plausible text, not to do that causal work, so a confident prediction from most AI ad tools is a guess dressed up as a forecast.
Why is a highly accurate AI tool potentially more dangerous than an unreliable one?
People build safety checks around tools they see fail often, but stop watching tools that almost never fail. That removes oversight right when a rare, costly error finally happens, and at scale those rare errors become statistically likely, not just theoretically possible.
What should a small business ask an AI ad vendor before trusting its claims?
Ask what the model's forecasts missed last quarter and how the team found out. A vendor doing real measurement has a validation process: tracking forecasts against actual results and running controlled tests. No answer to that question usually means no real measurement underneath the claim.