The forking paths
A thousand labs test a pill that does nothing. How many publish that it works?
A thousand labs test a pill that does nothing, and each measures several outcomes before writing up the one that came up. The middle panel counts the labs that announce it works; the right panel asks the same labs to catch a pill that really helps. Three tabs apply three reporting rules to identical data.
Board: Decision-Making Under Uncertainty →Three panels and a slider. On a narrow screen the panels stack. Nothing animates or plays on its own.
try this Drag Outcomes each lab looks at from 1 to 20, then switch tabs and watch Labs that announce it works.
Mood, sleep, memory, reaction time: a study can measure many things about one pill. Researchers call each choice a researcher degree of freedom, and a paper usually reports only the path it took. The tabs below are three rules for which result gets printed. All three run on the same thousand data sets.
The pill does nothing. Each of 1,000 labs measures 10 outcomes and reports any one that reaches p < .05. How many labs report that the pill works?
The first labs, pill does nothing
Each row is one lab. A finding here is always a fluke.
- 1nothing foundp = .159
- 2nothing foundp = .108
- 3nothing foundp = .218
- 4nothing foundp = .050
- 5nothing foundp = .101
- 6nothing foundp = .186
- 7nothing foundp = .067
- 8"works", on outcome 8p = .011
- 9nothing foundp = .163
- 10nothing foundp = .205
- 11"works", on outcome 2p = .016
- 12nothing foundp = .177
- …and 988 more labs: 392 of them also "found" it
Labs that announce it works
The arithmetic predicts 40%. The 5% everybody assumes holds only for one planned look.
If the pill really helps
Same labs and same noise, but now the pill shifts outcome 1 by a modest amount. One planned look catches it about half the time.
Why report whatever came up lands where it does
Each lab measures several things about the same pill and writes up whichever one cleared p < .05. Each single look has a one in twenty chance of a fluke, and the chances stack: with ten looks, about four labs in ten find something in a pill that does nothing. Nobody lied. Every number in every paper is real. The rule for choosing which number to print is what broke.
The tradeoff: This rule costs nothing to follow and feels like thoroughness. It pays twice: false findings climb with every extra look, and when the pill really works, the lab often headlines a louder fluke instead of the outcome that actually moved.
Five percent is a promise about one look
"p < .05" means one thing. If the pill does nothing, a single planned test comes out this extreme by chance about one time in twenty. It says nothing about a paper whose authors tried ten things and printed the one that worked. Each look is another ticket in the same raffle.
The chances multiply. The chance that ten honest looks all come up empty is 0.95 multiplied by itself ten times, about 0.60. So about 40% of labs see at least one fluke. At twenty looks it is about 64%. The slider shows the simulation landing on that curve.
Nobody has to cheat. Measuring more than one thing is normal, and so is deciding afterwards which analysis looks most sensible. In 2011 three psychologists used exactly these choices to "show" that listening to a Beatles song made students younger. They reported that combining four common choices pushed the false-alarm rate above 60%.
The fixes are not free. Raising the bar for every look keeps the false alarms down and loses real effects. Writing the plan down first keeps both numbers honest and gives up the surprises. That trade is why the field moved to preregistration rather than to ever-stricter thresholds.
Limits of this model
The outcomes here are independent. Real outcomes are often related (sleep and mood move together), so ten real looks inflate the false rate less than ten independent ones. This lab shows the upper end. The direction is the same either way.
The real effect is a teaching value. It is sized so one honest test catches it about half the time. Real effects in psychology vary, and many are smaller, which makes the cost of the stricter bar larger, not smaller.
A thousand labs is a sample, not the formula. The panels count real simulated labs, so they wobble a point or two around the arithmetic printed beside them. The noise is drawn from a fixed hash, so every visit and every tab sees the same thousand labs.
This lab does not model publication. In the real record, journals mostly printed the labs that found something and the rest stayed in a drawer. That filter makes the published share of false findings worse than any number on this page.