By Dr. Ken Broda Bahm:
I came across a recent article in the publication Raw Story with the intriguing title, “There’s a big problem in opinion polling that mainstream media is missing” by Matt Robison. While not an academic piece, the article does address a problem that is driving academics crazy. I’ve noted in previous posts the now-familiar problem that, even with a behavior as concrete as voting, pollsters are increasingly finding it harder to find a representative pool, stick to a reasonable margin of error, and produce results that match reality. Robison quotes a meta-analysis of public opinion research (Berinsky, 2017) that “there is no agreement among political scientists on how to best measure public opinion through polls,” also making the pithy observation that “to speak with precision of public opinion is a task not unlike coming to grips with the Holy Ghost.”
Sampling is one problem creating this uncertainty, but the “big problem” that Robison notes is to be found in the questions themselves and the fact that they often miss the nuance of what people are thinking when they answer. He notes, “We’re not measuring what Americans actually think or how they will act politically with anything near the accuracy that we believe we are.” The problem is that the results are clouded by variable context. Small differences in associated meaning provided either in the questions or by the people answering the questions will lead to very different answers. This matters when it comes to asking attitudinal questions in jury selection, both in questionnaires and in oral voir dire. Ask them whether there are “too many lawsuits,” for example, and the answer will depend on what kinds of lawsuits they have in mind. In this post, I’ll explain this context problem and offer a reasonable fix that trial lawyers should apply.
The Problem With Uncertain Context
Matt Robison gives a few good illustrations of the ways that responses can vary significantly based on contextual cues that may be unknown. For example, a question might ask about support for marijuana legalization, and a person could say “Yes” because they oppose overcrowded and underfunded prisons, not because they support recreational use of the drug.
He also cites the example of a recent question used by Pew Research. When the survey offered the simple statement that “Jobs are easy to find” in the location where respondents live, the results are a close to even split in “Yes” and “No” responses. But if you simply added the language of “good jobs” rather than “jobs,” the response pattern turned into a 27-point advantage for those saying “No.”
Even when questions are framed with some precision, like “Do you favor or oppose providing a way for undocumented immigrants already in the United States to become citizens?” it could matter whether we are we thinking of software engineers from Norway, refugees from Haiti, farm workers from Mexico, or simply the undifferentiated stereotype that could pop into the mind with the phrase “undocumented immigrants.”
Finally, the author notes that labels like “Medicare for All” or the “Green New Deal” generally boast strong support, but that support will often drain away when context is supplied. For example, when pollsters clarified that universal Medicare is the same as a single-payer health system, or highlight the role of federal spending in promoting environmental initiatives, the answers were different. The support “wobbled if brushed with a light feather of context.”
The Solution When Asking Jurors
Reading these examples, it occurred to me that broad open-ended questions with uncertain context are often exactly the kinds of questions that attorneys ask when questioning and selecting juries. Think of some of the more common question categories across cases:
-
-
- Are there too many lawsuits?
- Can corporations be trusted?
- Should we emphasize more individual responsibility?
- Do we need greater safety?
-
It is likely that the context that jurors would bring to those broad questions would play a big role in determining the responses, and as a result, the relationship between general attitudes and case-relevant views is probably more fragile than we would like to believe.
The other problem in litigation is that attorneys will often believe that they need to entirely avoid context in order to steer clear of asking jurors, or appearing to ask jurors, to prejudge the case during voir dire. Clearly, you can’t ask a potential juror, “Here’s the situation…now, how do you think you might rule in a case like that?” But that is pretty much what attorneys would like to know. To bolster our ability to reliably interpret responses, I believe providing some context – realistically, as much as the judge allows you to – will make for better and more useful answers.
For example, I have seen some attorneys wait until the end of their examination before offering the “This is a case about….” and “Hearing that, does anyone see any reason you couldn’t be fair in evaluating a situation like that?” questions. Rather than treating that as a clean-up step, you are far better off putting that question as close to the start as possible. By beginning with that inquiry, either at the top of the questionnaire or the start of oral voir dire, you can effectively encourage jurors to apply that as the context when answering general attitudinal questions through the rest of the questionnaire or the rest of oral voir dire. While general national pollsters often don’t want a specific setting to influence the response, in litigation you do want to know what they think in the specific context of the story at the heart of your case.
Of course, there is still likely to be quite a bit of variability to the ways different people understand just about any question. But when your polling mission is as specific as it is in litigation, the more context you can add, the better.
______________
Other Posts on Jury Questionnaires:
- Get Better Answers: Top 7 Posts on Supplemental Juror Questionnaires
- Juror Questionnaires: Don’t Worry (So Much) About ‘Helping the Other Side’