Advertising research doesn't replicate as often as gurus pretend.

In 1993, two psychologists played Mozart's Sonata for Two Pianos in D Major to 36 college students before a spatial reasoning test. The students scored a little higher right after listening than they did without the music. That was the whole study. One page, in the journal Nature.

Within a few years, Georgia and Tennessee had passed laws distributing classical music CDs to new parents. A toy and media industry built around the idea grew into what Atticus Li's rundown of the replication crisis puts at roughly half a billion dollars. The effect, when later researchers looked closely, turned out to be small, short-lived, and arguably just a byproduct of the music putting people in a slightly better mood before any test, not a specific "Mozart" effect at all.

That gap between what a study actually showed and what the world did with it is worth sitting with, because the same gap exists in advertising advice. Someone cites a famous finding with total confidence. The finding is real, in the sense that it was published. Whether it still holds up is a separate question, and it's the one most pitches skip.

The replication crisis is bigger than one bad study.

Psychology has a name for this problem, and it's not new. Psychology Today's summary of the replication crisis traces the term to a widely doubted 2010 paper claiming evidence for precognition, which pushed the field to start seriously checking its own work. What it found was not reassuring. A landmark effort by the Open Science Collaboration to redo 97 published findings succeeded on fewer than 40 percent of them. A separate 2018 replication project, the Many Labs 2 effort, testing 28 older findings from 1970 to 2014 found evidence for only about half.

The numbers aren't uniformly bad, which is its own useful data point. A separate 2018 project published in Nature Human Behaviour, checking 21 findings from top-tier journals like Nature and Science, found two-thirds replicated. Quality of the original venue matters. But even the best showing here means a third of findings from the most rigorous journals didn't hold up on a second look, and that's before anyone waters the claim down for a sales deck.

Priming effects, the idea that a small thing you see briefly, like a word or image, quietly changes how you behave without you noticing, and power-pose research, the claim that standing a certain way for two minutes changes your hormones and confidence, are both named in that Psychology Today summary as effects that drew heavy skepticism after failed replications. Both spent years as staples of conference talks and business books before the retesting caught up. Catchy ideas can circulate for years before anyone checks whether they actually hold up, which is exactly why a confident pitch deserves the same question as a dusty journal article: what was this built on? Your own ad deserves that same scrutiny before you trust it with your budget, and an outside read on your ad is what Knowing is built to give.

A failed replication isn't always proof the original was fake.

It would be easy to read all this as "don't trust any research," and that's the wrong lesson. Eisend, Franke, and Leigh's 2016 paper on reinquiries in advertising research pushes back directly on the idea that a real effect should produce identical results every time it's tested. The authors report that exact replication is a misunderstood ideal even in science generally, and that advertising research's inconsistent findings across studies are an expected feature of how knowledge accumulates, not necessarily a sign the original work failed.

A finding failing to repeat exactly doesn't mean it was fake. It can mean the effect only shows up under conditions nobody wrote down.

Their point: samples differ, time periods differ, media environments differ, and an effect found in one context can genuinely weaken or vanish in another without either study being wrong. That's a real and important distinction from fraud or sloppy work. It's also, inconveniently, a distinction that's easy to abuse. "The context was different" can be an honest explanation or a convenient excuse, and from the outside, a business owner reading a case study has no way to tell which.

The Stanford Prison Experiment shows how far a thin study can travel.

Nowhere is the gap between evidence and legend wider than the Stanford Prison Experiment. For fifty years it was the go-to proof that ordinary people turn cruel the moment you hand them a uniform and authority. It ran on 24 participants and was published in a low-tier criminology journal, according to a summary on Atticus Li's blog covering the replication crisis, which cites the original published study. Archival audio surfaced in 2018 and documented by French sociologist Thibault Le Texier's paper in American Psychologist revealed that the guards had been actively coached toward cruelty, not simply left to their own devices. A redesigned version of the setup, the 2006 BBC Prison Study run by Reicher and Haslam, found that guards did not slide into abuse on their own; it took deliberate changes in group structure and framing to produce any shift in behavior at all, a result that complicates Zimbardo's original story rather than simply reversing it.

Twenty-four participants and a low-scrutiny publishing venue shaped how institutions, schools, and even HR departments thought about human behavior under authority for half a century. Nobody fact-checked that against the sample size, because the story was good and the finding was useful to a lot of people's existing arguments.

Advertising research has its own version of this problem.

Advertising and marketing research isn't exempt from the pattern, and in some corners it's got less scrutiny than psychology does, not more, partly because advertising has no equivalent of a large-scale, independent replication project checking its claims the way psychology now does. The Interactive Advertising Bureau's (IAB) own evaluation of methods used to assess internet ad effectiveness exists precisely because measuring whether an ad worked is harder and messier than most people assume, which is also why the IAB separately published a document of best practices for conducting online ad effectiveness research, an admission that a lot of studies in this space aren't built carefully enough to trust at face value.

The field also has a habit of conflating things that aren't the same, which is its own quiet source of bad advice. Da Silva and Mazzon's 2016 paper on social marketing reports that the field suffers what they call an identity crisis, with cause-related marketing, corporate social responsibility, and plain institutional advertising all getting lumped together under one label despite having different goals and different ways of being measured. When the underlying concept is fuzzy, a study claiming to prove something about it is standing on sand before anyone even runs the numbers.

What to actually do with a confident claim.

None of this means ignore research or distrust every framework with a researcher's name attached. Binet and Field's work on long-term versus short-term advertising effects, for instance, has been checked against the IPA Databank, a large archive of real campaign results built from UK advertising effectiveness award submissions, not a single lab study with three dozen students. That's a different category of evidence, and it deserves a different level of trust.

The useful habit is asking a simple question before acting on advice delivered with certainty: what was this built on? A large dataset of real campaigns, checked more than once, earns more weight than a single small study that's never been retested. A famous name attached to a famous finding is not, by itself, evidence of anything except that the finding was famous.

The Mozart Effect didn't fail because someone lied. It failed because a one-page note with 36 students got treated like settled science, and nobody in the resulting half-billion-dollar industry stopped to ask whether the original evidence could carry the weight being put on it. Before the next pitch deck cites a study to justify its price, ask the same question.