Protzko et al. 2023 Nat Hum Behav was a high-profile preregistered paper on replicability in psychology. The headline result was a demonstration that you can get replicable results in psychology by using rigorous practices: pre-registering your study design and analysis plan; having a giant sample size; etc.
I say it “was” a high-profile paper because it has been retracted. It was retracted because it’s not ok to say your paper demonstrates the value of preregistration when it wasn’t actually pre-registered and instead was p-hacked (!) You know it’s bad when the journal not only retracts your paper, but lets two of your critics write a “Matters Arising” without publishing a reply from you.
Protzko et al. have been invited to submit a new ms for peer review, I assume on what was supposed to be the original topic of the study. Wonder how that new ms will go over with reviewers. [grabs-popcorn-meme]
Your mileage may vary as to whether you see this as a good news story, or a bad news story.
On the one hand, is it good news that the authors got caught trying to pass off a non-preregistered study as preregistered? The system worked, justice was done and seen to be done, etc. On the other hand, is it bad news that the authors–some of whom are high-profile preregistration advocates–tried to pull this off in the first place? After all, if even preregistration advocates can’t be bothered to do preregistration properly, why should the rest of us bother trying?
On the third hand, maybe it’s actually good news that the authors tried to pull this off in the first place? Because it shows that preregistration has now come of age, at least in psychology. Preregistration is now sufficiently valued that you can get a high-profile paper (at least briefly) by falsely claiming to have preregistered your work! That’s a sign of progress, right? I mean, not the sign of progress that preregistration advocates were hoping for, obviously. But a sign nevertheless. If preregistration weren’t widely valued, nobody would even try to get a paper in Nature Human Behaviour out of falsely claiming to have done a preregistered study. It’s like, you don’t want a drought to end with a flood, but the flood is a reliable sign that the drought is over.
On the fourth hand, maybe the fact that preregistration is widely valued in psychology is itself a bad sign for the field. Because it shows that the field has merely replaced one cargo cult with another. See this blog post from statistician Jessica Hullman. She contributed to the journal’s investigation of Protzko et al., and recommended retraction. Her experience dealing with this situation has changed her mind on the whole open science/preregistration/etc. movement in psychology. Her concluding lines:
Preregister if you find it helpful. Make your materials open because you should. But don’t expect these practices to transform your results into solid science, and don’t trust people that try to tell you it’s as easy as adopting a few simple rituals. I’m now doubtful that the flurry of research on fixing the so-called replication crisis is truly interested in engaging deeply with concepts like statistical power or replicability. I’m left wondering how many other empirical pro-open science papers are rhetorical feats to “keep up the momentum” regardless of what can actually be concluded from the data.
See also this detailed blow-by-blow blog post by Joe Bak-Coleman. Bak-Coleman also was involved in the investigation. He’s blunter than Jessica Hullman, and accuses Protzko et al. of lying to cover up their actions. Sample quote to give you the flavor:
I held out some shred of belief that people who have built careers on honesty, transparency, and openness could be honest, open, and transparent. I was naive.
I’ll conclude with a hopeful postscript. If you want to wash the bad taste of this story out of your brain, go read our old post on Gould et al., an admirably well-done preregistered study of “researcher degrees of freedom” in ecology. The comment thread is super-interesting. It includes thoughtful comments from Tim Parker, one of the leaders of the nascent preregistration/open science movement in ecology. I think the field of ecology is lucky that our own nascent preregistration/open science movement is led by thoughtful, careful scientists, rather than people who are looking to start, or jump on, a bandwagon.
p.s. Andrew Gelman comments too, but much of his post summarizes Hullman’s and Bak-Coleman’s posts.