"Psychology Findings Not Strong as Claimed", by Benedict Carey in The New York Times
To be clear, this replication study of 100 psychology studies did not reexamine their data and determine that the results were overstated; rather, it re-ran the experiments and found that 35 studies turned out the same, 62 produced the same outcomes, correlations, tendencies etc. but not as strongly as the original studies, and 3 were not counted because it was unclear how they turned out.
The stronger the results in the original study, the more likely they were to remain as strong when redone. The experience and prestige of the authors did not correlate positively or negatively with consistency of results.
The Times story also quotes some criticism of the replicators' methods and questioning of their rigor. But it does seem significant that 62% of retests showed weaker results, and apparently none showed stronger results,* and none showed completely different results, either.*
This certainly does seem to show some kind of tendency to overstate results, although it is hard to see how, since they are based directly on the data. And it appears that that tendency kicks in when the results are not that overwhelming in the first place. Or perhaps there are other factors that make the results stronger the first time an experiment is run and published? Or maybe results that are not so overwhelmingly one-sided are more likely to vary in different samples?
Perhaps more significant is that the experiments practically always ended up with the same conclusion, just not as strongly.*
"Social science isn't rocket science, it's harder", Edward Tufte said when sharing the article on Facebook. But it's important to remember that this study was really designed for medical sciences, and was applied here to psychology studies. However, many social sciences, especially demographics, do not involve those kinds of studies, and sometimes reproducing results is not a possibility in many social science fields. While psychology studies are a significant share of the items on this blog, many more are demographic studies, polls, or in other disciplines of the social sciences.
__________
*Items marked with * are what I assume from the Times article, but it does not say so as explicitly as I would hope. I didn't look at the original study it reports on. I hope someone will do so, and post a comment saying whether these interpretations of the article are correct.