Publication Bias and the Study Hype Cycle
Takeaway Points:
Certain types of studies are more likely to be published than others. This skews the data by hiding studies and data which are less interesting or which have negative or neutral results.
Likewise, studies funded by organizations with a financial interest in those studies can be skewed in order to produce the results that organization wants.
If the data doesn’t suggest a particular outcome, it’s possible to reorganize the data to provide (weak) evidence for a different outcome. This produces exciting data and is more likely to result in being published, but is poor science.
The scientific method suggests that if we can only get a particular outcome once, it’s an outlier and not the general rule. Likewise, if a study is performed only once and with a small sample size, the data it provides may be an outlier. Unfortunately, most studies are never repeated because this is more boring than funding new studies.
Popular reporting of science in the media compounds this issue by boiling complex science down to attention grabbing headlines.
Calculations enable us to estimate the chances that a particular study is likely to be affected by publication bias, and thus the chances that it would be overturned by more data. The bigger the sample size and the more studies saying the same thing, the less likely it is that this data could be overturned by the revelation of future “hidden” data.
The study hype cycle.
Science isn’t perfect. There are plenty of ways in which flawed studies can be published. That being said, one of the big problems with quality of research goes along with the way that the public expects studies to act, resulting in real effects on what kinds of studies do get published, and what kinds of studies don't.
While we generally expect science to focus on giving us (sometimes boring) answers to important questions, some scientific studies are more “interesting” than others. If there’s a new exercise fad, supplement, or dietary plan on the market, there’s pressure to study these things because they’re new and current.
Scientific journals are more likely to publish papers on, and talk about, things that are exciting to the general public - even when their general public is supposed to be disinterested and neutral.
Likewise, this gives scientists reasons to seek out “interesting” findings in their papers. This is known as publication bias. If your findings aren't interesting, you may not get published at all, and that invisible "hidden data" can hide the real conclusions you should be reaching.
Let’s say that you wanted to study the effect of a specific supplement on athletic performance. You may expect it to improve strength, or increase muscle mass or endurance. So you perform a study, but the data basically says nothing - there’s no significant effect. Essentially, the paper says that this supplement probably doesn’t do anything.
But this is a boring result - it’s not interesting. A publication is less likely to publish this, because it means that you haven’t figured anything out. Yes, you gathered important data, but you weren’t able to draw any strong and exciting conclusions. So as a result, scientists are often pressured to spin existing data, or seek out new data, that shows interesting and significant results.
Worse, let’s say that you wanted to prove that the supplement works, but this study was funded by a supplement company with a vested interest in this supplement. In this case, as a researcher you may be pressured to spin the data in a way that provides a positive or neutral result.
Spinning existing data involves searching through the data for any potentially significant finding, even if it’s likely not to be a correct one. Say in the same example, you were looking for evidence that this supplement improves your strength, but instead you crunch and fiddle with the data until you find a significant effect for… stomach pain. Insert anything you want in here: the point is that if you can find any evidence of the supplement doing anything, no matter how weak that evidence may be, you can pat yourself on the back for having discovered something new. Off to the presses!
Or, if there’s really no way to “salvage” the data, they may simply refuse to publish the data at all, effectively making it vanish, since this means that they can continue to sell the ineffective supplement. If you don’t tell anyone about the data you gathered, no one will know about it. It's like the sleazy car salesman who tries to sell you a junker but doesn't bother telling you that it's probably just gonna break down in a week.
Science is founded on the idea of repeatability. If you take a specific pill and get stronger, you might assume it's the pill that did it, but you have to repeat that action a few times to make sure that you’re consistently getting the same result - that this gain in strength wasn't an outlier, caused by something else. Maybe there’s an alternate universe where that pill doesn’t make you stronger nine times out of ten, and it’ll only do so one time in ten. But in our universe, we can confirm that the pill works only because we take it repeatedly (or have a lot of people take it) until we have a lot of data points to show that yes, this is what happens when you take this specific pill.
This is a problem when they have small sample sizes, (when they’re conducted on a small number of people) because more data is needed. Sometimes, when studies are repeated, new data or different results are revealed. If a study cannot be repeated with similar results, this may mean that its findings are outliers and therefore not accurate.
Unfortunately, many scientific studies are performed only once. Another aspect of publication bias is that there’s little motivation to repeat existing studies to get more data. While more data would make it clearer what’s actually going on, it would also require more money and time to gather it, and it isn’t likely to give us any shocking and new results since we've already gotten a rough picture through the first study. Thus, it’s boring - and less likely to get funded or published.
Popular reporting of science in the media compounds this issue by boiling complex science down to attention grabbing headlines.
In one particularly telling example, a journalist purposefully made up a (mostly) fake study claiming that eating chocolate helps you lose weight. He was just trying to poke fun at journalists. This fiction came with a compelling premise (chocolate aids in weight loss) and, as expected, it created a media frenzy.
This was the epitome of purposefully bad science, but because it was attention getting, it became widely and uncritically reported. Worse: since many readers of these news publications never realized he was pulling a prank, lots of them probably continue to believe that chocolate causes weight loss!
Since journalists are less versed in science than scientists themselves, they tend to misinterpret studies, exaggerate publication bias, and give overly confident interpretations of weak or uninteresting data.
In practical usage, you can use methods to determine how likely it is that additional data would overpower the current data. If you have a lot of data on a subject (huge sample sizes or meta-analysis), it’s less likely that an unpublished or unrepeated study would mess everything up. If you have only a small amount of data, however, it’s more likely. Calculations give us a rough idea of how likely additional “hidden” or “unknown” data would be to overturn the current strength of evidence, and this can be used to judge how likely it is that a certain scientific conclusion is the correct one.
Of course - most of this is completely unknown by most looking to understand scientific papers, who will, like the media, uncritically accept any evidence at face value.
Further Reading:
Are you interested in perfecting your deadlift and building legendary strength and muscle? Check out my free ebook, Deadlift Every Day.
Interested in coaching? Inquire here. You can also subscribe to my mailing list, which gets you the free GAINS exercise program for maximizing strength, size, and endurance.
