How to Detect Bullshit in the Scientific Literature – Spin edition

Scientific papers are not immune to bullshit.

If we’re looking for an answer to a question, we can never trust just one paper. We have to see if the findings have been replicated (ideally many times and by different people with different biases) and that the methods were sound before we can conclude anything.

That makes the process of science slow and tedious, but that’s why we love it. Mostly we don’t have answers, but when we do, they come after decades (sometimes centuries or millennia) of scrutiny.

Therefore, when reading a single paper, or even a review of many papers, we gotta be critical of what we’re reading. This is especially true in an environment where many authors try to make their findings seem novel in order to get published.

It’s also especially important if the findings seem to confirm our preexisting beliefs.

For this edition of bullshit-detecting, I’m going to talk about “spin” in a science paper. We always have to watch for a discrepancy between what the evidence says and how the authors try to spin results to make it look like they’ve found something new and exciting.

We all know this from politicians, but it’s prevalent in every field, and it’s a big problem in the primary-research literature.

---

 

Let’s look at one paper as an emblematic example of this issue.

Our question is, “what are the best and safest treatments for lower-back pain?”

To tackle this problem, the American College of Physicians recently released new guidelines for treating lower back-pain.

In the U.S., lower-back pain afflicts millions of people every year, can be really expensive (estimated to rack up billions of dollars annually), and can seriously disrupt people’s lives.

These guidelines purport to be evidence-driven, which is great, because we need solutions to dealing with pain conditions (I’m especially excited as someone with chronic stomach problems).

Basically, the authors based the guidelines on recent systematic reviews and a few controlled trials looking at pharmacological and non-pharmacological treatments for lower-back pain. They amassed the evidence to paint a bigger picture for what works and what doesn’t. It’s a good idea and we need guidelines like these to help physicians give patients the best evidence-based treatments.

BUT, watch for spin.

---

 

 Let’s start with the narrative of the paper, and what the authors recommend for treatments:

1. For acute or sub-acute pain, the authors strongly recommend: superficial heat, massage, acupuncture, spinal manipulation (an umbrella term for a variety of treatments), and if desired, NSAIDs (like ibuprofen) or muscle relaxants.
2. For people suffering with chronic pain, the authors strongly recommend: exercise, multi-disciplinary rehab (which is another umbrella term for a variety of treatments combining physical exercise with psychological and social interventions) , acupuncture, mindfulness-based stress reduction, tai-chi, yoga, relaxation, laser therapy, cognitive behavioral therapy, and spinal manipulation.
3. Only for patients with chronic pain who did not benefit from noninvasive treatments, the authors (weakly this time) recommend: prescribing anti-inflammatory drugs, antidepressants, and opioids as a last resort.

First of all, I am happy to see opioids being taken down a peg. The authors sensibly write,

“Clinicians should only consider opioids as an option in patients who have failed the aforementioned treatments and only if the potential benefits outweigh the risks for individual patients and after a discussion of known risks and realistic benefits with patients.”

This is a much-needed change. Over-prescribed opioids continue to contribute to the epidemic currently claiming tens of thousands of lives every year. So, using them only when absolutely necessary seems not only reasonable, but long-overdue.

But what about the other treatments?

Before we even look at the data, let’s start with some obvious spin. The first thing that should pop out at you is the lack of clear definitions for each treatment.

For example, the authors differentiate between tai-chi, yoga, and exercise. Why? They also differentiate between mindfulness-based stress reduction and relaxation. Why are the two different? Why these separations without a clear reasoning for the definitions?

This is spin when you see that distinguishing between modalities here actually makes no sense. The authors, and most of the papers they cite, do not do a proper job of isolating yoga or mindfulness as distinct therapies from exercise and relaxation.

The few papers that do compare between modalities actually find no difference. For example, a paper they cite to support the recommendation for yoga actually finds no difference between yoga and stretching. This makes sense: how is stretching in yoga different than stretching without yoga? As for relaxation, some of the papers they themselves cite find that different but similar therapies like mindfulness and cognitive behavioral therapy have the exact same effect on alleviating pain.

This is not about dismissing mindfulness or yoga as effective (we haven’t looked at the data yet). It’s about spin. If, for all medical intents and purposes, yoga is a form of exercise, and if meditation is a form of relaxation, then why differentiate?

This is the kind of spin that tries to dress up findings in a trendy or sexy way. It’s problematic mostly because it’s sloppy science: if we were comparing two forms of stretching and found no difference, why call one Y and the other X? This kind of spin might also be an issue since it may inadvertently insert faith-based or magical practices into medicine.

---

Now let’s look at the actual data.

The biggest tool for detecting bullshit in a research paper is to see whether the evidence actually supports the conclusions of the authors.

In Table 1 and Table 2 in the appendix, you’ll find how a treatment is graded as compared to placebo or no treatment (no effect, small, moderate, or inconsistent results). You’ll also find how the evidence is graded and how many studies it’s based on (low quality or moderate quality evidence, based on a number of studies, usually randomized-controlled studies)

The basic rule of reading the table is this: a treatment should only be considered effective IF you have enough high quality studies with enough people studied, accounting for bias in methods, that suggest the treatment is effective at alleviating or getting rid of lower-back pain.

It’s also crucial that the treatments be compared to placebo and not simply to nothing. For example, if an opioid did better than no treatment, but a sugar-pill performed as well as the opioid, we would say the opioid is NOT an effective treatment.

Then we have to look at the effect of the treatment and compare to the quality of the evidence:

A small effect from moderate evidence in a placebo-controlled trial is good.

A moderate effect from a low-quality trial or one without placebo control is bullshit.

And no effect is, obviously, no effect.

So, what works?

• Pain-relievers like ibuprofen, muscle-relaxants, opioids, and steroids are all pharmacological treatments with a small effect for pain and function (with tramadol having a bigger effect)
• Exercise, massage, multidisciplinary therapy, and heat-wraps are all non-pharmacological treatments with a small effect (with heat-wraps having a bigger effect)
• Mindfulness therapy has a vague “improved” effect

 

That’s it. That’s what works.

Notice that yoga, tai-chi, spinal manipulation, laser-therapy, cognitive-behavioral therapy, relaxation, and the other modalities either show no effect or are not supported by good enough evidence.

Acupuncture is an even more interesting case. The acupuncture vs. sham (placebo) studies either don’t show an effect or are based on very low-quality evidence. It’s only when acupuncture is compared to nothing that it has an effect. In other words, it’s placebo, just like a sugar-pill.

Now, I don’t want to be unfair. The authors do point out where treatments are backed by low-quality or moderate-quality evidence, and they do discuss some of the shortcomings of the data:

“The evidence is also insufficient for most physical modalities. Evidence is insufficient on which patients are likely to benefit from which specific therapy.”

Most importantly, their first recommendation clearly points out that:

“most patients with acute or subacute low back pain improve over time regardless of treatment”

And lastly, their emphasis on not taking pharmacological drugs unless absolutely necessary is refreshing, and we should see more of that.

Basically this is how I’d sum up the findings: most patients will improve with time no matter what they do, and for most of the treatments examined there is no clear evidence of benefit. The best advice is to stay active with exercise, relaxation, occasionally put heat on the area, and drugs but only if the pain really persists.

The authors do state those things.

And yet, the authors DO also recommend (strongly recommend) a whole host of unfounded treatments (like yoga, acupuncture, and spinal manipulation).

This is incredible. Their abstract and recommendations directly contradict the data and their own words. 

This would be fine for a company marketing these treatments. It is not OK in an evidence-based recommendation of guidelines for physicians across the country.

It’s spin. It’s misleading. It’s bullshit.