Friday, May 24, 2013

Should media-worthy psychology research be held to a higher standard of evidence?

I've been attending the APS meeting in Washington DC, and this afternoon had 3 back-to-back sessions on research best practices (5 straight hours without a break!). The sessions included extensive discussion of the issues of p-hacking, small samples, and other biases. It focused not just on problems, but also on possible solutions, and it ended with a set of brief commentaries from a panel of editors and program directors. In all, it was a fascinating session.

For me, perhaps the most interesting and provocative suggestion came from Danny Kahneman who argued that studies that are likely to be of broad public interest should be held to a higher evidentiary standard because they are more likely to influence the public perception of our entire field. He agreed with many of the other panelists in arguing that most studies should have much bigger samples (i.e., higher power to measure effects precisely), but argued that media worthy studies should be held to an even higher standard: Journal editors should insiste that such studies be highly powered and highly likely to be replicable. He also argued that journal editors and societies must enforce these standards, perhaps establishing guidelines that are implemented over a period of several years much like government requirements give car manufacturers some years to implement gas mileage standards in their fleet.

Should media worthy studies be held to a higher evidentiary standard, or should all studies be required to meet the same standards? Would imposing a higher standard for media-worthy studies stifle risky, novel, or innovative research, or would it help to improve such research (or both)? Let's hear your thoughts in the comments.

Update & ClarificationA few people seem to be misunderstanding my question. I'm not asking whether the media should be held to a higher reporting standard. That's a different topic. Rather, I'm asking whether journal editors should hold scientific manuscripts to a higher standard of evidence if the results reported in the manuscript are likely to be media worthy once they are published.