Spin, Hype and Replication Problems in Scientific Research

in #science6 years ago (edited)

Scientific research gets published into many journals from a variety of fields. There are papers for "hard" or natural sciences[1] that are concerned with the natural world like biology, chemistry, physics and astronomy. These are considered to apply a more "pure" form of the scientific method compared to the "softer" sciences related to social life[2] like psychology, sociology, economics, political science and history.

There are some issues with both of these categories of scientific research. Spin and replicability issues can arise that cast doubt and put results into question. Spin is normally thought of as the domain of news media, politicians and propaganda where biased presentation of information is used to get a certain message accepted. Scientific literature isn't immune to using similar practices.


publicdomainpictures.net/CC0 Public Domain

When it comes to biology and biomedical research there is a high level of distortion that creeps up in research papers. Authors and researchers use certain methods to make the data appear more favorable in order to validate a certain conclusion or result. This leads to misleading the readers of these scientific papers.[3]

'Spin' or 'science hype' is a problem as it "can negatively impact the development of further studies, clinical practice, and health policies".[4] Further research can then be directed into areas that actually lack evidence that merit it, as well as increase financial investments into useless or harmful applications.

Researchers at University of Sydney Charles Perkins Centre and Faculty of Pharmacy set out to determine how pervasive spin was by conducting a meta-research study of other meta-research studies. They conducted a systemic review of 34 other reviews/reports that had analyzed spin in "clinical trials, observational studies, diagnostic accuracy studies, systematic reviews, and meta-analyses".[4].

"Half of the reports (17/34, 50%) that assessed spin in published literature assessed spin in both the abstract and main text, 4 of which specifically compared the main text results to the abstract and/or main text conclusions as a measure of discordance. Suggesting that the consequences of spin in the abstract were more severe given that many clinicians rely on abstracts alone, 7 reports (7/34, 21%) assessed spin in the abstract only. Nine reports (9/34, 26%) assessed spin only in the main text of the article. Three reports (3/34, 9%) additionally assessed spin in the articles' titles."[4]

The most prevalence of spin was found in research about non-randomized trials. The meta-research reports (34) had 84% spin tactics used in the main text of the studies looked at, and 65% used in the abstracts alone.


Table 3[4]/Creative Commons Attribution (CC BY)

The studies looked at in the 34 reports showed an average of 30% (141/474 of studies analyzed) in abstracts and 22% (75/346 of studies analyzed) in main text that contained "high" levels of spin in their conclusions. A high level of spin in conclusions is defined by Boutron et al.[6] as:

  • "no uncertainty in the framing of conclusions"
  • "no recommendations for further trials"
  • "no acknowledgment of the statistically nonsignificant primary outcomes"
  • "making recommendations to use the intervention in clinical practice"

Spin was most commonly performed through selective reporting of only statistically significant data and distorting "statistically nonsignificant results" like stating something "shows an effect". This can lead to an overemphasis on certain data that overshadows other data or outcomes. There is also the use of causal language to mistakingly attribute causality such as suggesting "X leads to Y", "X increases Y", or "X facilitates the rapid recovery of Y". Tone inferences such as "this study shows that" or "the results demonstrate" is another factor.[5]

When spin is used, researchers can give the impression that claims are supported by the evidence when they aren't exactly. Certain data can be favorably presented or other adverse data can be underreported to produce optimistic abstracts. Clinical practices can be take research finding as recommendations for them to pursue.[3]

Conflicts of interest and funding sources were looked at in 19 of the 35 meta-research studies associated with spin, but no clear conclusions could be drawn from hypothesizing these factors as the source of distortions and spin. It's possible research incentives or rewards contribute to the construction of positive conclusions in order to gain media attention or to be accepted for publishing.[3].

Lead author, Kellia Chiu, said "[t]he scientific academic community would benefit from the development of tools that help us effectively identify spin and ensure accurate and impartial portrayal and interpretation of results."

In addition to initial research that is presented through spin or hype, even if the research is without these influence, there is also a problem of being able to replicate findings. A 2015 study[8] tried to revalidate the proposed cause and effect of some research by reproducing 100 experiments. The alleged "statistically significant" effects found in 2/3 of the original studies could not be replicated.

It was done on psychological studies, but other fields of research like cancer studies also fail this basic check of replicability. There are incentives to publish research which shows a causal link ("X leads to Y", "X has an effect on Y"). This biases researches to find effects. when experiments are done, researchers can be motivated to report on favorable effects that prove a hypothesis, and under-report others that would nullify their hypothesis. Data is farmed and cherry-picked to confirm results that are desired. But the finding may not be repeatable, as is often the case. [7]

This behavior of distorting data is often encouraged by those funding the research who usually want a certain result. Sensational findings can get more rewards compared to others which make more modest claims. The media latches onto the hype of certain stories. Researchers are thus encouraged to issue press releases to try to garner more attention and popularity which can lead to more funding. A lot of funding in universities is allocated through a numbers game. The more publications are made, the more funding can tend to be provided, regardless of the content. This encourages simply pumping out papers to be published. [7] Journals are often more interested in publishing papers with positive conclusions which means others that establish negative or inconclusive findings can get pushed to the bottom of a wait list for approval.

One effort to make research more honest is to remove incentives for publishing by accepting to publish research before any experiment is run. The goal is to reduce farming for data and just present it as is, no matter the outcome. Statistical competency is lax in some cases as well. And what makes things worse, is that some researchers don't release the statistical methods and data which means others can't verify what they did exactly, and replication is harder to do. Some publishers and editors don't have a properly defined statistical methodology, and mistakingly access flawed statistical data used to conclude findings. The Open Science Framework is a good way to get data reviewed before publishing one's work and avoid spreading incorrect information.

These might help to some degree in mitigating the replication crisis, but the real solution is for scientists to change their mindsets. This would require admitting that a conclusion can be true or false, not blindly standing their ground and trying to force it to be what they want it to be. Curiosity, uncertainty and doubt are a scientists friend.


pixabay/CC0 Creative Commons

When publishers don't know how to detect flawed science, this can result in hoaxes getting published. Or when it's a pseudo-journal, people can accept a study without knowing they are bing fooled. Take the case of Gary Lewis from Royal Holloway, University of London. He announced on his Twitter account that he "submitted a hoax manuscript to a predatory journal. The finding? Politicians from the right wipe their ass with their left hand (and vice versa) - big breakthrough! Manuscript accepted w/o review".

His intentional-hoax paper concludes the following:

"The descriptive statistics showed a clear pattern. Politicians of the right were more likely to wipe their bottoms with their left hand (4 out of 4). The opposite pattern was seen for politicians of the left, with 3 of 4 wiping their bottoms with the right hand (Jeremiah Doorbin responded that he used a munchkin from The Sound of Music to do the wiping, but intimated that if did the wiping it would depend on which hand was free at the time). Using structural equation modeling us formally confirmed this finding – the AIC was 1654.23 and the RMSEA was .02. These are excellent fit statistics although the model makes little sense."[9]

Notice the part about a munchkin from the Sound of Music. The flagrant ridiculousness and falseness of this "study" is obvious, and there have been others like The Conceptual Penis, but some "studies" aren't so obvious. Different studies can contradict each other, purporting different conclusions, so they can't both be true. Yet they are published as science or "truth". At least that's how it's played out in the media and how most people take it when the media says it's so. And some studies snowball one after the other for years, only later to be proven to be false, sometimes uncovering dubious funding sources that pushed for false conclusions to be drawn.

Sometimes even when there isn't spin used, it can still be hyped up by the media who don't read a study properly and come to the wrong conclusion themselves. The hype of a publication can still be spread despite all efforts made by a researcher to present the data honestly. It would be great if the problems of replication were resolved so that studies could more easily be verified and bad science could be stopped before moving ahead with further funding or wasted resources.

If researchers stopped using spin to hype up their findings, that would also improve the reliability of scientific papers. Despite science not being able to control media, scientists can take it upon themselves to be more vigilant within their field and apply better standards as individuals and as a collective workings towards the honest discovery of what 'is' in reality.

With these potential misrepresentations of the data, it doesn't mean the conclusions are necessarily wrong even if worded with distortions. Many -- if not most -- studies are accurate overall. Progress is made through research. If all the studies had false conclusions, nothing would move forward in science.


References:
[1]: Hard and soft science
[2]: Social science
[3]: Watch out for hype—science 'spin' prevalent, researchers warn
[4]: Chiu K, Grundy Q, Bero L (2017) 'Spin' in published biomedical literature: A methodological systematic review. PLoS Biol 15(9). DOI: 10.1371/journal.pbio.2002173
[5]: Lazarus C, Haneef R, Ravaud P, Boutron I. Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention. BMC Med Res Methodology. 2015;15:85. DOI: 10.1186/s12874-015-0079-x
[6]: Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058–64. DOI: 10.1001/jama.2010.651
[7]: The Replication Crisis in Science
[8]: Estimating the reproducibility of psychological science. Science 28 Aug 2015, Vol. 349, Issue 6251: DOI: 10.1126/science.aac4716
[9]: Gerry Jay Loui. Testing Inter-hemispheric Social Priming Theory in a Sample of Professional Politicians-A Brief Report. Psychology and Psychotherapy. June 15, 2018.


Thank you for your time and attention. Peace.


If you appreciate and value the content, please consider: Upvoting, Sharing or Reblogging below.
Follow me for more content to come!


My goal is to share knowledge, truth and moral understanding in order to help change the world for the better. If you appreciate and value what I do, please consider supporting me as a Steem Witness by voting for me at the bottom of the Witness page.

Sort:  

As a reader and researcher who has done experiments I have to say honestly that in truth less than 10% of the abstracts prove nothing more than fraud. The truth here is that so called scientist are putting food on the table over honest science. That is those who are ignorant of the scientism occurring within the field.

Those who know are complicit in the crimes and those who don't know are ignorant. Does not bod well for reputation and is why Science as a field is loosing it's reputation. That I think is the reason for the fraud. To destroy the reputation of Science as a field of study.

Sadly, the way they have dumbed down education while turning science/medicine into a religion of sorts for the dumbed down masses I disagree it is destroying the reputation for most. I have read many of your posts for example, and there are many topics you talk of that I am guessing would get you a lot of blank looks if mentioned in large circles of people. Same with the posts @krnel writes. Most prefer to let the "experts" do the thinking for them.

Glad you are back @practicalthought and you are right. My own family think I am an alien from mars or something. That is what happens to a person when they self examine for the express purpose of self improvement as I am sure you know. They also tell me that I am more lovable everyday. Guess aliens have something going for them anyway.

There are a couple of post on @aconsciousness and @commonlaw that many think I am crazy for even now. What is cool about that is I have learned to know I am going in the right directions when they project such emotions at me. The truth really is stranger than fiction! LOL

Yup, honest science is hard to propagate when so much bias and compromise to methodology is done in order to serve agenda.

That is exactly it @krnel it is being bought and paid for instead of discovered.

"Researcher's Bias" is almost always going to be an issue. I always start seeing red flags when someone opens with something like "we set out to prove that white vinegar kills XYZ viruses."

OK... you've already predetermined what you want the outcome to be, so now your methodology, sampling and wording will automatically skew in that direction. And that's before we even get to spin. "Lying with facts" is a time honored tradition, sadly... because everyone has an investment in their opinion; nothing is left to simply be information.

Well said. Confirmation bias, we need to invade Iraq, so lets saw Iraq has WMDs, now lets prove it, even though we can't prove, but we'll say we proved it, and then everyone will believe we did even though we didn't.

Good read. There is too much bias in science. Lots of scientists and researchers don’t truly employ the scientific method and go into a research situation and identify their biases and be wary of them.
There’s also the refusal of science to be scientists in many situations; if a thousand scientists look at a topic and come to a similar conclusion and 5 new ones challenge the preconceived notion that the others state they are ridiculed. As a scientist you are supposed to constantly look at information and analyze it for improvement.

Yup. That seems to be the case for many things. Climate change (formerly known as global warming) not caused by humans? Nah. It's 100% proven science. Those are just climate-deniers ... :/

That may be where we disagree perhaps; I am aware that we as humans certainly have an effect on the climate of the planet. That I don’t have trouble with, what I have trouble with is the amount of effect we have. I don’t think it’s as cataclysmic as they want you to believe. It goes along the lines of keeping people scared and fearful when I think of how they hype it up. I personally think that we are to be blamed for maybe 15-20% of the way it’s changing; the planet is quite old and has gone through many changes in its time. We simply do not have the knowledge to say. They can look at tree rings, mud samples, Arctic ice samples but it remains that we did not have the recording devices that we do now back then. It’s speculation as to the exact degree of what happens.
Back to my previous comment, there is no 100% proven science. The entire basis of the scientific method is to reanalyze data continuously to look for changes and new understandings. If we just say, as is far too often the case, that it’s done then we fail as a scientific community. Break throughs and revelations happen when people challenged the norms of the day. Today is no different, we need to be continually challenging ourselves by putting aside our preconceived notions and think objectively.
Just my opinion of course, but I see it all too often with the major sciences of today and it’s a shame, we can’t learn as a society if we aren’t willing to uncover a falsity or half truth that we all accepted because that’s what we were taught/told/indoctrinated to.

It's possible research incentives or rewards contribute to the construction of positive conclusions in order to gain media attention or to be accepted for publishing

I have always viewed this as a given. The researches will most often sell out for the funding and recognition, the projects they are assigned ones seeking to prove what those with the money want them to prove. I would go further and say if the evidence shows the opposite, it is buried and careers as well if those researching don't go along with the program.

Yes, in many cases there is a lot of pressure to comply with agendas rather than speak the truth of the findings :/

when it comes to biology and biomedical research there is a high level of distortion that creeps up research papers. Authors and researchers use certain methods to make the data appear more favorable in order to validate a certain conclusion or result.

I noticed that a while back with some research papers I read on climate change and human sexuality.

I think the major problem is funding and social acceptability like you rightly observed. Dissent is hardly tolerated, even in the research World

Yup, stick with the mainstream direction, or risk being an outcast :/

The politicization of science is a far worse problem than most people would credit. I am glad to see you dealing with issues regarding credibility in science. The pressure to publish papers that attract funding is phenomenal. 'Publish or die.'

Thanks!

Yup, there is a lot of pressure, and getting the $ means more than being an honest scientist :/

the field of educational research is the worst for this. they would have studies that were self report surveys of fewer than 10 kids and they would suggest actual policy changes based on that! And then by the time some asshole at the New York Times tries to understand it and hype and politicize it then all of the original research is practically pointless. Like when Goddard published his research about rockets working in space in 1920 and the NYT wrote an editorial saying he didn't understand Newton's laws and was an idiot.

Curated for #informationwar (by @openparadigm)
Relevance; Science is a useful tool not an authority of truth.

  • Our purpose is to encourage posts discussing Information War, Propaganda, Disinformation and other false narratives. We currently have over 7,500 Steem Power and 20+ people following the curation trail to support our mission.

  • Join our discord and chat with 150+ fellow Informationwar Activists.

  • Connect with fellow Informationwar writers in our Roll Call! InformationWar - Contributing Writers/Supporters: Roll Call Pt 8

Ways you can help the @informationwar

  • Upvote this comment.
  • Delegate Steem Power. 25 SP 50 SP 100 SP
  • Join the curation trail here.
  • Tutorials on all ways to support us and useful resources here

Excellent analysis. Science is being used to support many corporative crimes. Importantly, I like how you define your goal. You deserve real success. -- Congratulations and thanks for this article.

Thanks ;)

Coin Marketplace

STEEM 0.35
TRX 0.12
JST 0.039
BTC 69796.92
ETH 3521.66
USDT 1.00
SBD 4.70