Today’s post will shed some light on how this may not be so – and lift a veil on some of the problems and controversies that surround research studies today. For this, I spoke with Andrew Gelman, a professor of statistics and political science at Columbia University and a father of three children: Zachary (12), Jacob (10), and Sophie (5). Andrew has been an outspoken critic of various practices that some researchers use, intentionally and not intentionally, to “find” their research findings.
It's not so much that successful studies get published and unsuccessful studies get buried. Rather, there's a pressure for successful studies, and just about any study can be made publishable via a careful enough inspection of the data to look for patterns.
Suppose you get people to roll a die 600 times and hypothesize that of all six faces, some are more likely to come up than others. The problem here is that we have many different things to look at: we could compare the 1's to the 6's, or compare low numbers to high numbers, or compare even numbers to odd numbers, or look at some other combination such as comparing men who roll even numbers to women who roll odd numbers. We could have a story for any of these. Any specific reported claim typically represents only one of many analyses that could have been performed on a dataset.
And researchers can test (but not disclose) any number of these combinations until they find a significant p-value. Could you explain what you and others have called the "replication crisis" in science?
Published and publicized research claims that disappear when independent groups try to replicate the studies themselves. There have been lots of well publicized examples, including empowered cognition and power pose (just google for background on these if you'd like).
You are referring to the study that showed that power posing can change your hormone levels – which drew a lot of attention but hasn't been replicated by other scientists. You have suggested that the finding may have been spurious. But if power posing helps a lot of people (e.g., at job interviews), why does it matter that the study results haven't been replicated?
You ask, "If power posing helps a lot of people." That's a big "if." What makes you think so? It could be that power pose is hurting lots of people. Maybe job candidates are practicing their power pose instead of practicing yoga and meditation, which might actually work? Maybe they're practicing power pose instead of practicing answering interview questions. As far as I can tell, the only evidence of the effectiveness of power pose are some unreplicated studies and a bunch of anecdotal evidence, with the anecdotes collected by supporters of power pose. It could just as well be that it hurts, or has no effect, or helps some people and hurts others.
If the proponents of power pose want to give talks and write self-help books, fine, go for it. Just be accurate about what your evidence is, and don't claim that a study with many uncontrolled “researcher degrees of freedom” tells you anything at all.
And it's not just about seeing patterns in tea leaves, or silly psychology studies. I'm afraid we do this same sort of reasoning all the time, seeing patterns from noise in everyday life. When our children were little, we were always trying to figure out how to help them get to sleep. For a while we focused on regular bath times, then there were the different variants of swaddling, the favorite stuffed animals, and so forth. We'd try something new and it would work--but were we just lucky? That said, there was no harm in us experimenting.
So, sure, if the two alternatives are: (a) Try nothing until you have definitive proof, or (b) Try lots of things and see what works for you, then I'd go with option b. But, again, be open about your evidence, or lack thereof. If power pose is worth a shot, then I think people might just as well try contractive anti-power-poses as well. And then if the recommendation is to just try different things and see what works for you, that's fine but then don't claim you have scientific evidence one particular intervention when you don't.
Pattern-finding and intuition, valuable as they are in suggesting new ideas, can mislead if we naively overstate the evidence in favor of our latest theories. It's a problem for us as we make our daily decisions and also bedevils trained scientists. We can't expect scientists to clean up their act all at once but I would like them to be more aware when they can be wrong.