Open Science and the Certainty of Bias

This past year has been quite exciting for publishing in the domain of chiropractic. This is not because of any ground-breaking discoveries, although some very respectable research has been published, but rather because of the vigour of debate. I am thinking in particular of discussions around 2 papers, one concerning the application of spinal manipulation to the management of non-musculoskeletal disorders1, and the other a rapid review of the use of x-ray in chiropractic practice2. In both instances, rebuttals to the original publications made reference to biases not just in the methodologies, but in the values of the authors. The implication, therefore, was that the conclusions were unjustified because the authors had reached the conclusions before performing the studies. This would be a very pointed critique, and begins to look like a personal attack on the characters of the researchers. Further, some would suggest this was perhaps because there was less to criticize about the research activity itself. Happily, the debate was mostly on a pretty civilized plane.

Publicly critiquing researchers’ motivations or biases does take us into dangerous territory and can distract from the more forensic task of analyzing the research methods, results and conclusions. Indeed, there is a whole ‘open science’ movement around transparency, beginning with the founding of scientific journals and really codified in the middle of the last century3.  These days, open science is especially focused on the content of the research publication. Hence, researchers are encouraged, and sometimes compelled, to publish their methods in advance of actually performing an experiment, and to make their raw data available through publicly accessible repositories. There are all sorts of real-world benefits to this kind of open science, including facilitating the more cost-effective of replication of studies, and accelerating innovation by making methodologies available prior to publication of peer-reviewed articles.

The idea of critiquing scientists themselves, however, is fraught with danger. On a broader scale, it may fuel unjustified scepticism about the whole enterprise of science and so feed into the narratives of conspiracy theorists and other species of fruitcake. At the level of the individual, it is just plain hurtful and seldom more than speculative. Nonetheless, the idea of the scientist as a dispassionate and humble servant of their discipline is something that needs to be challenged. We are not without biases and we almost certainly bring those biases into our professional work.

As an example of this, the career researcher almost certainly has made the judgement that their area of investigation is worthwhile. If I am studying back pain it is because I have made the judgement that is an important area of study – more important than some other complaint, such as turf-toe. In this example, there may be some objective support for the judgement: 92 hits in Pubmed for turf toe versus ~68,000 for back pain; but it is not always so clear. In the 2 more provocative examples of research cited above, the authors may have had well-grounded concerns about the inappropriate use of spinal manipulation or x-ray. Should that exclude them from conducting research in these domains? If we were to apply that standard broadly, one wonders how much research would survive.

I therefore recently enjoyed reading an article by Kevin Elliot in which he proposed a taxonomy for transparency which could be applied to the values that we carry into our research4. I like the idea because he does not propose that we open our souls with every publication, but that there should be a framework for declaring a priori values and judgements according to the intent and audience for each research paper. Scientists do have values, and they should have values, as these are important drivers of research, and Elliot’s paper is a useful place to begin open discussion of how our personal values impact our research.

References:

  1. Côté P, Hartvigsen J, Axén I et al. The Global Summit on the efficacy and effectiveness of spinal manipulative therapy for the prevention and treatment of non-musculoskeletal disorders: a systematic review of the literature. Chiropr Man Therap. 2021; 29: 8.
  2. Corso M, Cancelliere C, Mior S, Kumar V, Smith A, Côté P. The clinical utility of routine spinal radiographs by chiropractors: a rapid review of the literature. Chiropr Man Therap. 2020 Jul 9;28(1):33.
  3. Merton, Robert K. 1979 [1942]. “The Normative Structure of Science.” In The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press. ISBN 978-0-226-52092-6. 
  4. Elliott, K. C. 2020. A Taxonomy of Transparency in Science. Canadian Journal of Philosophy: 1–14, doi:10.1017/can.2020.21

Leave a Comment

Your email address will not be published. Required fields are marked *