By Dr. Ken Broda Bahm:
As a twice-weekly blog writer focusing on the social science of legal persuasion, I am thankful that during this medical emergency, the research has kept coming. Even as restaurants, bars, beaches, and schools have been shut down, the researchers have continued to turn in their research. In some ways, it has even picked up a bit, with new studies on pandemic-related public attitudes coming out almost daily. And the pace of medical research has been even more robust. A recent story on NPR reported, that just in the last couple of months, more than 6,000 scholarly papers have emerged on the coronavirus. Many to most became available as “preprints” shared online, largely bypassing the prepublication processes of peer review that are designed to ensure quality and consistency with academic norms.
The NPR story quotes Theo Bloom, the executive editor of the medical journal BMJ (formerly the British Medical Journal): “As human beings, I think we default to thinking: ‘How do I know this, where does it come from, who’s telling me and do I believe them.'” In other words, we are drawn to some of the surface cues relating to the source of the conclusions, and tend to focus less on how they got to their conclusions. But that is a shortcut. And in science, shortcuts can have some pretty negative results. Aside from the tide of coronavirus scholarship, there is one other context where research often bypasses the constraints of peer review: scientific testimony that is part of the litigation process. While judges, counsel, and jurors will often focus on the conclusions and the source, a full and fair evaluation requires looking at the methods. When you believe that your own expert fares better on the methods than their expert, you will want to emphasize this method-driven approach. In this post, I will share three messages that your expert witness can sponsor in encouraging jurors to avoid the shortcut, look beyond the source and the headline, and to dig into how the conclusions were obtained.
Not Infallible, but Systematically Fallible
This can be something that is hard for people to understand if they don’t work in or with science, but it is not supposed to be infallible. Instead, it is intended to work within a system that makes any flaws known in short order. For example, we have probably heard of early studies on using hydroxychloroquine to treat coronavirus, and several, both pro- and anti-, were later retracted. Some of the retractions even came from vaunted sources like The Lancet and The New England Journal of Medicine. But these retractions don’t show that the process is crooked, or that we fundamentally cannot answer these questions. Rather, they show that in some ways the research process is working as it should. When a study comes out, there are many other researchers who are motivated to closely study the data and the methods and to bring attention to problems if they exist.
This emphasis on the fallible and self-correcting nature of science could be put to productive use in testimony, with the message being:
You don’t need to believe that the opposing expert is lying, you just need to understand that all science can have flaws, and that discovering and working past those flaws is part of the scientific process.
It’s About the Method (Not the Source or the Conclusions)
Working past the flaws means paying close attention to research methods. Theo Bloom, the BMJ editor, notes that researchers looking at the coronavirus studies haven’t been able to rely on the standard cues relating to the source due to the fact that many of these studies are originating in China or other Asian nations where the virus first made itself felt, and involve less familiar laboratories and lesser known publications. As a result, evaluators need to focus more on the methodology and the original data quality. And that is a good thing. “There are retractions and falsifications from great journals, great institutions, from Nobel laureates and so on,” she notes, “So, it behooves us all to try and move away from who we recognize as good,” and to instead focus on the research process.
The message on this theme from an expert in litigation could be as follows:
Ultimately, it does not matter much or at all what our names are, or where we went to school, or how many studies we have published. What matters is what we did in this case to reach the conclusions that we are offering.
Parachutes Do Actually Work
A study published in Bloom’s journal, BMJ (Yeh et al., 2018) reached the surprising conclusion that, “parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention.” And it wasn’t just a think-piece, it was based on a randomized study where volunteers actually jumped from a plane or a helicopter, after being randomly assigned to “parachute” or “no parachute” conditions. Measuring the resulting deaths and injuries, the study concluded that there is no advantage to a parachute. In fact, the analysis showed no statistically significant differences between the parachute and no-parachute groups when it came to deaths or injuries.
How could that be? Close readers discovered that it was because both the plane and the helicopter were parked on the ground when the individuals jumped. After all, how else would they have gotten volunteers for the study? The descent was about two feet for all of the study participants. The authors, of course, were making an important point in a lighthearted manner. The moral is that a study’s conclusions are only as good as its methods. We need to look beyond the broad conclusions and beyond the trappings of the scientific method and ask how it is done.
I firmly believe that including this example would be an entertaining part of direct examination for your expert. If it helps you to emphasize the importance of methods and not just conclusions, then have your expert illustrate it to the jury using the parachute study. It could be a memorable way of making the point that, in a way, the opposing expert stacked the results and, in effect, left the aircraft on the ground as well.
____
Other Posts on Scientific Testimony:
- Adapt Your Scientific Testimony to Jurors’ Skeptical Ears
- Measure Your CSI Effect
- Experts, Tell a Story
Yeh, R. W., Valsdottir, L. R., Yeh, M. W., Shen, C., Kramer, D. B., Strom, J. B., … & Nallamothu, B. K. (2018). Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial. bmj, 363.