Wednesday, May 1, 2013

A step too far: What replications do and do not imply

An article that appearing in Nature yesterday focused on a recent paper in PLoS ONE that reported a failure to replicate a classic social priming study by Ap Dijksterhuis. Unfortunately, the headline and thrust of that Nature piece incorrectly linked a failed replication attempt to the noted case of Stapel's fraud. It inappropriately impugned an entire discipline (social psychology) and country (the Netherlands) based on a single paper that was unable to replicate a few related priming effects. That's not just a step, but an entire staircase too far.

Replication failures do NOT indicate fraud. They are a part of the normal scientific process. Even if the original effect is completely true, some proportion of studies will fail to replicate it. And, even if it is false, one negative result does not prove that. Replications can fail for many reasons (and spurious positive results can occur as well). Replications like the PLoS paper add evidence. Perhaps the original effect is a little weaker than previously thought. Or, perhaps it is more sensitive to subtle differences in method. Or, perhaps the replication attempts were not done correctly. Or, perhaps the original study was not done correctly. Only with multiple, independent, direct replications can we better estimate the true underlying effect. A single result, positive or negative, does not provide definitive evidence. It is informative. Just not that informative.

I spoke the author of the Nature piece for 20 minutes when she was researching it. I didn't comment on the latest replication controversy. Instead, I talked with her about how psychology is taking the lead on addressing replicability and best practices. I talked about the great new initiatives at APS and other journals, and the changing incentives that will promote publication of direct replication attempts. Those changing incentives will improve the state of our science going forward, and they might generalize to other fields as well. Unfortunately, none of those positive comments made it into the piece.

If you want to read more, I'd recommend what  had to say about the controversy.

Update: Fixed some formatting.

Update 2: I fixed the typo in the title of the post (to -> too) almost immediately after publishing, but blogger won't let me fix the link. I guess it's permanent. Blech.


  1. Hey! I know this is somewhat off topic but I was wondering which blog platform are you using for this site? I'm getting tired of Wordpress because I've had issues with hackers and I'm looking at alternatives for another platform. I would be awesome if you could point me in the direction of a good platform. comprehensive SEO campaign

  2. Hey! This is my first visit to your blog! We are a collection of volunteers and starting a new initiative in a community in the same niche. Your blog provided us valuable information to work on. You have done a wonderful job! handy digital marketing tool

  3. For Top Influencers Marketing In Singapore, you should get in touch with Media One. The agency has been providing you with the right influencer marketing techniques and strategies. They would ensure that your brand has been made prominent through influence marketing.

  4. Great beat ! I wish to apprentice while you amend your web site, how could i subscribe for a blog web site? The account aided me a acceptable deal. I had been a little bit acquainted of this your broadcast offered bright clear seo singapore

  5. Among the popular options made available online, you should look forward to MediaOne. You should invest in MediaOne to hire a good SEO company to help you optimise your business. It has been largely popular for all kinds of SEO needs in the Singapore region.