Literature Review – How to Choose Content
The overarching goal of this project is to communicate the scientific process and the methods attached to that process. Therefore, of course, deciding where to start my research was easy: my first topic was what is science, why do it, the history of the scientific method, and the multiple infrastructures currently in place that govern and facilitate the work of millions of people around the world that we laypeople call “the scientific community” (Raptor 2015; Wikipedia 2016b; Harari 2014; Prescod-Weinstein 2015; Byrne 2016).
It was and is very clear, however, that discussing the basics would not be enough to touch on the important subject of how to critically assess evidence and its reliability. No. That requires delving into the limits and faults of science. It requires delving into current topics that are important to most people, like the environment, healthcare, racism, etc.
That left me with the question of how to choose topics in a systematic way so I am not simply cherry-picking things I care about without regard to public interest or public service.
I decided which topics I would like to discuss based on where the available literature indicated that science communication/education is failing (i.e. where public distrust is most rampant), where conspiracy theories abound, and where there are valid and important criticisms of science research (Goertzel 2010; Nattrass 2013; Novella 2002; Harding 1986; Wikipedia 2016a; Seth 2009; Adas 2008; Goldacre 2013b; Goldacre 2013a; Funk and Alper 2015; Funk and Rainie 2015; Prescod-Weinstein 2015).
I landed on these topics:
- The scientific method, its history, its value, and its limitations
- This had to include cognitive biases and logical fallacies and the ways in which scientific methods are meant to safeguard against things like motivated reasoning or confirmation bias
- Colonial history in science
- Much of science delegitimization occurs in the context of legitimate grievances stemming from the often brutal and horrific connection between imperialist/colonial regimes and scientific research or technology
- Bias and implicit bias in science (including racism in science history and current practices, discrimination and underrepresentation of people who are nonwhite and not male in scientific fields both in academia or industry, and gender bias in scientific history and currently)
- For-profit motives in science and how they influence research
- Science communication and science education (mostly in the United States)
- This includes the skeptical literature
- Integrative, complementary, or alternative medicine
- Naturopathy (this is an umbrella term that includes many specific practices like acupuncture, homeopathy, prescribing “natural” medicine or supplements, and can sometimes also include “energetic healing” practices like reiki)
- Chiropractic (highly variable field but based often in pseudoscience, debunked, or untested practices)
- Popular pseudoscientific beliefs, or unfounded beliefs that are either overtly anti-science, or that have been shown to empower uncritical thinking, feed distrust of the scientific method and scientists in general, and/or lead to conspiracy thinking
- Vaccination fear/skepticism/denial
- Evolution denial
- Climate change denial
- HIV/AIDS denial
- Fear of GMOs
- “Natural” food fads and chemical illiteracy
- Believers (or sometimes con-artists) like Psychics, Fortune-tellers, Intuitive Healers, and Astrologers that often give unsubstantiated medical/life advice
- Some of these are obviously harmless or fun. However, there is evidence of some overlap from one field into a more harmful field like opposing vaccinations, denying HIV/AIDs, etc (Goertzel 2010; Oliver and Wood 2014; Chigwedere et al. 2008).
Literature Review – Search Methodology
For each topic I conducted search combing the literature for a designated 60 hours through the Cochrane Library, PubMed, PLoS/PLoS ONE, GoogleScholar, Nature, Science, BMC, and BMJ databases.
Search terms included (slashes indicate different search):
- “[topic name, e.g. “naturopathy/homeopathy/MMR vaccine”] systematic review/meta-analysis/review/consensus/state of the science”
- “[topic name] efficacy/effectiveness/evidence/evidence-based”
- “[topic name] cover-up/conspiracy/hoax/lie(s)/problems/issues”
- “[topic name]” on its own
Some topics are more context-dependent, of course, so searches were modified. For example, for HIV/AIDS denialism in addition to the above search terms I also searched “HIV/AIDS link/connection”. For information on vaccines I searched “vaccine [this or that vaccine, or in general] safety/autism/danger/risk”.
Source Reliability and Cataloging
All sources were cross-checked with Wikipedia, Scholarly Open Access, Elsevier, PubMed, and SagePublications databases in order to check for legitimacy and relative impact factor.
Reliable and relevant sources were saved and cataloged by category into a Zotero© library named “Science-Based Thinking.” I will make that library publicly available for viewing for anybody interested. As of this writing (Dec. 15th, 2016), the library contained 247 sources.
*Note: For topics such as CAM (complementary and alternative medicine) peer-reviewed sources from CAM journals were also deemed “reliable” and saved into Zotero©, regardless of the methodological failings of many studies. This was done in order to engage seriously with the evidence provided by advocates of such practices and to actively guard against putting myself in an echo-chamber
Hierarchy of Evidence
Systematic reviews (ideally from multiple unconnected sources with limited industry interests) were prioritized over clinical trials, in vitro studies, or preliminary case-studies. Studies with double-blind and/or randomized methodologies were prioritized over studies without such safeguards against bias.
The reason to use such a hierarchy of evidence is well articulated by the Skeptical Raptor:
“instead of consciously or unconsciously (or deviously or innocently) choosing the papers that support your point of view, you use science on science. The systematic review examines all of the evidence, attempts to determine which evidence is weaker (based on statistical results and experimental design which might influence the results), and finally assembles it all into one study of studies that is complete and representative of all the research that has been done in one specialized field” (Raptor 2015).
Of course, study design including sample size and blinding protocols, conflicts of interest, and other factors like the use of statistical “cheating” (like multiple comparisons or p-hacking in its myriad forms) was evaluated to determine the validity of studies. This is all in keeping with current accepted standards of evidence (Moher et al. 2015; Shea et al. 2007).
Blogs and Websites
For each topic, I also conducted the same searches in anti-science websites and science communicators/skeptics websites. The anti-science websites are labelled as such because they have been shown to regularly contain misinformation, false information, and/or conspiracy theories (Pomeroy 2016).
I consulted these sites for a few reasons. First, in order to understand how pseudoscientific arguments/logic function and to get to know the people who either advocate for or buy into these positions. It was also to make sure I didn’t become entrenched in an opinion and closed myself off to evidence that may contradict my biases/opinions.
I consulted the skeptical literature/blogs in order to hear scientists’ and science-enthusiasts’ takes on the same issues, again with a critical eye, as well as to make sure I was aware of what had been done and learned by other skeptics so I was better prepared.
- Anti-Science sources: Answers in Genesis, Food Babe, NaturalNews, Mercola, Age of Autism, InfoWars, Rethinking AIDS, David Wolfe’s FB page, Breitbart, HuffPost* (*no just kidding about this one)
- Skeptic sources: The NESS, Science-Based Medicine, Skeptical Medicine, Skeptical-Science, Logic of Science, The Credible Hulk, Skeptical Raptor
Keeping Up on the News
To keep myself updated on current and relevant research I used Feedly© to set up alerts for the following sources and I comb through them once a week:
- Respected peer-reviewed research journals: Nature, Science, BMC, The various PLoS publications, PubMed, NIH, NEJM, The Lancet, PNAS, BMJ, Journal of Science Communication, (I also followed PeerJ though it might be of a lower standard)
- Peer-reviewed CAM (Complementary and Alternative Medicine) journals: BMC Complementary and Alternative Medicine, Evidence-Based Complementary and Alternative Medicine
- Peer-reviewed journals of meta-analyses and systematic reviews and scholarly-integrity watchdogs: RetractionWatch, Scholarly Open Access, Cochrane Library, BMC Systematic Reviews
- Weekly science stories as covered by the media: NYTimes Science, BBC Science, The Guardian Science, NPR Science, ScienceDaily, The Economist Science
I also consult the same list of anti-science and skeptical blogs at least once a week in order to keep myself open to opinions of all sides and to see what each community is saying or responding to.
These news searches are clearly less methodical than the literature review I conducted (as described above). My takes on news-items are therefore much more open to my own biases and what I find interesting. But you can be sure that I will only stake any kind of position if and only if I can base it on an existing and compelling body of evidence. For items where the questions are more open I will try not to voice an opinion and simply admit ignorance or uncertainty.
The Hierarchy of Posts
Full blog posts will only be written after a minimum of two/three-week period of independent research and consulting with other researchers, skeptics, and/or people who are distrustful of science.
Once a blog post is written, it will be edited at least twice by me and once the team grows by at least one other contributor or outside-eye for quality and bias checking.
This is NOT an echo-chamber. This is also NOT a twitter-feed. Every post has to go through scrutiny before going public.
This is done so you know that everything you’re reading is a labor of love and sweat meant only to provide you with information that strives to be as accurate and reliable as possible. Mistakes will happen. Oversight will happen.
If you spot a factual mistake or a misleading statement, please write. If you spot a citation of an unreliable source, or a misused citation, please write. This is NOT a soap-box. I am only striving to start honest, generous, and reality-based discussions about these subjects.
Articles posted from other sites/sources will be analyzed and fact-checked by me and as the team grows by at least one other person before being posted. These, however, will be less rigorous. The objective with these is either entertainment or to engage people to think critically about sources. Again, feedback and criticism will be most welcome.
*I will post the occasional meme or blurb. These are for fun, but I WILL NOT post anything that is misleading, incorrect, or highly partisan/ideological. Some of these may be more provocative or crass, but I’ll double-check them to make sure I’m not spreading misinformation.
Adas, Michael. 2008. “Colonialism and Science.” Encyclopaedia of the History of Science, Technology, and Medicine in Non-Western Cultures, 604–609.
Byrne, John. 2016. “What Is Science? – Skeptical Medicine.” Accessed December 11. http://www.skepticalmedicine.com/what-is-science.
Chigwedere, Pride, George R. Seage III, Sofia Gruskin, Tun-Hou Lee, and Max Essex. 2008. “Estimating the Lost Benefits of Antiretroviral Drug Use in South Africa.” JAIDS Journal of Acquired Immune Deficiency Syndromes 49 (4): 410–415.
Funk, Cary, and Becka A. Alper. 2015. “Religion and Science.” Pew Research Center: Internet, Science & Tech. October 22. http://www.pewinternet.org/2015/10/22/science-and-religion/.
Funk, Cary, and Lee Rainie. 2015. “Public and Scientists’ Views on Science and Society.” Pew Research Center: Internet, Science & Tech. January 29. http://www.pewinternet.org/2015/01/29/public-and-scientists-views-on-science-and-society/.
Goertzel, Ted. 2010. “Conspiracy Theories in Science.” EMBO Reports 11 (7): 493–99. doi:10.1038/embor.2010.84.
Goldacre, Ben. 2013a. Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients. Reprint edition. New York: Farrar, Straus and Giroux.
———. 2013b. Bad Pharma: How Medicine Is Broken, and How We Can Fix It. HarperCollins Publishers.
Harari, Yuval Noah. 2014. Sapiens: A Brief History of Humankind. Random House.
Harding, Sandra G. 1986. The Science Question in Feminism. Cornell University Press. https://books.google.es/books?hl=en&lr=&id=27TrCuk4LRgC&oi=fnd&pg=PA7&dq=sandra+harding+science&ots=qCmmBv7FYY&sig=DH5dXIH-ibGKL72yHD0lNT9ZV9M.
Moher, David, Larissa Shamseer, Mike Clarke, Davina Ghersi, Alessandro Liberati, Mark Petticrew, Paul Shekelle, and Lesley A. Stewart. 2015. “Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement.” Systematic Reviews 4: 1. doi:10.1186/2046-4053-4-1.
Nattrass, Nicoli. 2013. The AIDS Conspiracy: Science Fights Back. Columbia University Press. https://books.google.es/books?hl=en&lr=&id=almsAgAAQBAJ&oi=fnd&pg=PR9&dq=the+aids+conspiracy+science+fights+back&ots=QDb-XT-Rt9&sig=pyVajdPVq15q78She1PW5tfGWwg.
Novella, Steven. 2002. “Skepticism and Denial.” The New England Skeptical Society. April. http://www.theness.com/index.php/skepticism-and-denial/.
Oliver, J. Eric, and Thomas Wood. 2014. “Medical Conspiracy Theories and Health Behaviors in the United States.” JAMA Internal Medicine 174 (5): 817–18. doi:10.1001/jamainternmed.2014.190.
Pomeroy, Ross. 2016. “The Worst Websites for Science in 2016 | RealClearScience.” RealClearScience. November 28. http://www.realclearscience.com/blog/2016/11/28/the_worst_websites_for_science_in_2016.html.
Prescod-Weinstein, Chanda. 2015. “Decolonising Science Reading List – Chanda Prescod-Weinstein.” Medium. April 25. https://medium.com/@chanda/decolonising-science-reading-list-339fb773d51f#.cbb3lns13.
Raptor, The Original Skeptical. 2015. “Developing and Supporting a Scientific Consensus.” Skeptical Raptor. May 31. http://www.skepticalraptor.com/skepticalraptorblog.php/developing-supporting-scientific-consensus/.
Seth, Suman. 2009. “Putting Knowledge in Its Place: Science, Colonialism, and the Postcolonial.” Postcolonial Studies 12 (4): 373–388.
Shea, Beverley J., Jeremy M. Grimshaw, George A. Wells, Maarten Boers, Neil Andersson, Candyce Hamel, Ashley C. Porter, Peter Tugwell, David Moher, and Lex M. Bouter. 2007. “Development of AMSTAR: A Measurement Tool to Assess the Methodological Quality of Systematic Reviews.” BMC Medical Research Methodology 7: 10. doi:10.1186/1471-2288-7-10.
Wikipedia. 2016a. “Scientific Racism.” Wikipedia. https://en.wikipedia.org/w/index.php?title=Scientific_racism&oldid=748521537.
———. 2016b. “History of Scientific Method.” Wikipedia. https://en.wikipedia.org/w/index.php?title=History_of_scientific_method&oldid=751849099.