A small business owner now has a tool that can spit out fifty versions of the same ad before lunch. Fifty headlines, fifty photo crops, fifty ways of saying "book now." That part works. The part that doesn't work as well, and the part almost nobody talks about, is what happens next: picking which of the fifty is actually good enough to spend money on.
Those are two different jobs. One is generating options. The other is judging them. It's tempting to assume the same AI that wrote the fifty ads can also tell you which one is best, since it clearly understands the material. The research says that assumption is wrong, and it's wrong in a specific, measurable way worth understanding before you trust it with your ad budget.
Two jobs, not one.
The industry has started drawing a line between two kinds of AI: generative AI, which writes something when you ask it to, and agentic AI, which pursues a goal on its own over time, deciding what to do next and adjusting based on results. The CDP Institute puts it plainly: generative AI "focuses on content creation and synthesizing information," while agentic AI "focuses on decision-making and workflow automation." Most small business owners are only using generative AI, the writing kind, when they ask a chatbot to draft ad copy. The decision-making part, if it happens at all, is often handed to the same tool by default, because it's sitting right there and it sounds confident.
That confidence is the problem. A generative model writing your ad copy is doing a narrow, well-defined task: take a prompt, produce text or an image that matches it. Deciding which of fifty drafts will actually perform is a different task entirely. It requires comparing the new thing against a track record the model doesn't have unless you give it one. Without that history, picking a winner comes down to a guess dressed up as a decision, which is why Knowing judges an ad against the goal you set for it.
What happens when you ask AI to grade its own work.
A team led by Penn State marketing professor Wreetabrata Kar tested this directly, publishing their method in the Journal of Marketing Research. The setup, described in a write-up on Phys.org summarizing the published research, was straightforward: use AI to generate a large batch of marketing emails, then compare two ways of narrowing them down. One way scored the new drafts against a company's own historical campaign data, a kind of map built from what had actually worked before. The other way just asked the AI to pick its own five best emails.
The self-picked batch was predicted to perform worse. As the researchers put it, "when we asked the AI model to pick its own five best emails, those picks were predicted to do poorly." The model that wrote the drafts was not a reliable judge of which of its own drafts were good.
The model that wrote the ad is not automatically the best judge of whether the ad is any good.
The historical-data approach worked better because it measured similarity to proven past performance, not aesthetic judgment. The researchers' own example is useful: "'Free Shipping on Orders $50+' and 'Orders Over $50 Ship Free' end up side by side, while 'Break Free from the Rules' ends up far away, even though it also uses the word 'free.'" A simple word-count check would have grouped those messages differently than an approach that actually understands meaning. Messages too far from anything in the company's own history got flagged for a human to test the old-fashioned way: a small, controlled test, like an A/B send to a slice of the list, rather than being scored and trusted automatically.
A boutique owner uses AI to draft fifty Instagram captions for a fall sale. Instead of asking the same chatbot "which five are best," she checks which drafts sound closest to the captions that actually drove sales last fall, and sends anything wildly different from her history to a quick side-by-side test before spending money behind it.
Your own eyes aren't a reliable backstop either.
It's tempting to think the fix is simple: if the AI is a bad judge, a human should step in and eyeball the ad instead. That's true, but it's worth knowing how shaky even human judgment is here. A study out of UC Berkeley's Haas School of Business, by researchers Huang and Katona, had people guess whether an ad was made by a human or by AI. They got it right only 64% of the time, barely better than a coin flip, and their confidence in the guess had no relationship to whether they were actually correct.
The more interesting finding is what happened to the ratings. When people believed an ad was human-made, they rated it significantly higher across every effectiveness measure they tracked, even when that ad was actually made by AI. The belief about the ad's origin moved the score. The ad's actual origin, on most quality measures, didn't. The researchers found AI-made ads scored lower only on two specific things, creativity and memorability, and even those gaps shrank when people didn't know which was which.
So the honest picture is this: the AI that wrote your ad is a bad judge of its own work, and a person eyeballing the ad is also working off an unreliable signal, because what they think they're looking at changes the score more than what's actually there. Neither shortcut holds up on its own.
What this means for picking a draft.
None of this is an argument against using AI to write ads. Drafting fifty versions in the time it used to take to write one is a real gain, and no owner should feel guilty about using it. The argument is narrower: don't let the tool that wrote the draft also be the tool that tells you it's good.
The steadier approach, borrowing from Kar's method, is to judge new drafts against your own history, not against a fresh opinion generated on the spot. Look at what this audience actually responded to before. If a new draft is close to that, it's probably safe to run. If it's a wild departure, that's exactly the one worth testing carefully rather than trusting because it sounds clever or on-brand to whoever wrote it.
That's a structural fix, not a judgment call. It means building the judging step around evidence that existed before the ad did, whether that's your own sales data, documented research on how ads actually work, or a second, separate evaluation that isn't just the same model checking its own homework. The writing can be fast and cheap. The judging has to be slower and grounded in something the writer doesn't have access to about itself.
Frequently asked
Common questions.
Can AI tell if an ad was made by AI or by a human?
Not reliably. A Berkeley study found people correctly identified an ad's origin only 64% of the time, barely above chance, and their confidence didn't track with accuracy. Ads merely believed to be human-made scored higher on effectiveness, regardless of who actually made them.
Is it bad to let AI pick its own best ad draft?
Research from Penn State's Wreetabrata Kar found that when an AI model was asked to pick its own five best email drafts, those picks were predicted to perform worse than drafts screened against the company's actual historical campaign data. Generating and judging are separate skills.
What's a better way to screen AI-written ad drafts than asking AI which one is best?
Compare new drafts to what has actually worked for your business before, using your own past campaign results as the benchmark. Drafts far outside that history are the ones worth testing carefully, rather than trusting a model's self-assessment of its own writing.