Only 39 of 100 psychology studies replicated in 2015

In 2015 the Reproducibility Project: Psychology repeated 100 experiments published in leading psychology journals in 2008, and only about 39 were judged to have replicated. Reported in Science in August 2015, the effort enlisted more than 270 researchers, coordinated by Brian Nosek and the Center for Open Science, who reran the studies, working with the original authors where possible. Forty of the studies came from Psychological Science.

The gap shows up however you measure it. 97% of the original studies had reported statistically significant results; only 36% of the replications did. Even effects that held up were, on average, about half their original size.

The headline also hides a split: cognitive psychology findings were about twice as likely to replicate as social psychology findings, though both shrank by a similar amount. And 39% does not mean that 61% of psychology is false. A failed replication can reflect a real effect that is smaller than first reported, or one that only appears in certain settings.

Still, the benchmark is sobering: if every original effect had been genuine, at least 89% should have replicated. Pooling these replications with 207 others from later projects lifts the success rate to about 64%, with effects still around a third smaller than first claimed. The usual suspects are well known: p-hacking, publication bias against null results, small samples, and hypotheses written after the results were in.

What has changed since is the publishing system. Many journals now offer Registered Reports, which accept a study on the strength of its methods before the results exist, alongside preregistration, open-data badges and big multi-lab efforts such as Many Labs.