By Dr. Ken Broda-Bahm:
Your typical mock trial might involve three juries, with a total of 30 or so mock jurors. The typical public opinion poll run by an organization like Gallup, however, can involve more like 1000 participants. So what is the disconnect when it comes to sample size, the variable that researchers refer to as “N”? The answer is that public opinion surveys and mock trials/focus groups serve different purposes, and while there are definitely predictive benefits that come from researching with large samples, there are practical costs as well.
A recent article in DRI’s publication For the Defense, “The Case for Scientific Jury Experiments,“ addresses the sample size question head-on, and takes a side. Authors Christopher Robertson, David Yokum, and Bernard Chao — researchers from Universities of Denver and Arizona and the private sector — make the argument that instead of running traditional mock trials, consultants and litigators should be running large-N survey-style pretrial research on their cases. While the article makes a number of valid points, the choice is not as simple as they make it out to be. While you gain some forms of predictiveness in moving to that statistically valid sample size, you also lose some of the working value when you test with a broad survey rather than a traditional focus group or mock trial. In this post, I’ll take a look at what the argument gets right and wrong, and argue that there is room for both the large-N and the smaller-N methods, depending on your purpose.
What the ‘Big-N’ Argument Gets Right
The group’s DRI article is correct that there are limits on the usefulness of traditional focus group and mock trial based on the number of participants. With 30 to 50 participants, you cannot make statistically reliable predictions within a meaningful confidence interval. They’re also right that mock trials and focus groups do not necessarily identify the unconscious motivators that jurors are following. While you can look at your mock jurors’ behavior and preferences, when it comes to asking for the why, you can’t always trust their own self-diagnoses, since people won’t always know the causes of their own reactions. Finally, they are right that a larger sample-size survey can be reliably predictive (at least based on the limits of what is being tested — more on that below) and as a result, the research can be a better settlement tool (an earlier article from the same authors makes that argument).
You should consider large-sample research when your predictions on case reactions and valuation (based on a “thin slice” reaction to your case) is most important, or when you are needing a reliable answer to a very specific question (like the effect of offering a damages anchor).
What the ‘Big-N’ Argument Gets Wrong
The assumption behind the article, and the assumption that many skeptics of mock trials and focus groups will make, is that the main function of focus group and mock trial research is to predict case outcomes. It isn’t. Instead, a mock trial or a focus group is a way of working on your case. It is a way of getting human reactions to an informationally rich case presentation, while knowing these reactions are not statistically predictive, but are nonetheless part of the spectrum of what you would see in trial.
What mock trial critics also tend to downplay is the fact that a large-sample survey will nearly always require an extremely truncated version of the case. The authors in this case say it typically ends up being about 15 minutes per side and, in my experience, that is about the maximum for how much content you can meaningfully convey as part of a large-sample survey. In fifteen minutes you can share a fair amount, and there is a definite value to knowing what jurors’ reactions will be to just that “thin slice” of the case story. But compare that to hour or more per side of content that is typically conveyed in a traditional mock trial or focus group. The survey version will yield juror reactions to that thumbnail sketch, but you won’t get reactions to the specifics. You won’t learn how they react to the central documents, how they assess the credibility of the key witnesses, or whether they buy your specific answers to the other side’s arguments.
Finally, the authors criticize the group discussion that is part of focus groups and mock trials. They say the interaction can be an echo chamber, or can conversely end up polarizing the group. That is true enough, but that interaction is often precisely what you want to see. In the real jury’s deliberations, it doesn’t come down to individual responses, it comes out of group discussion. What you want to know is the product of interaction. What arguments will become part of the discussion and what arguments will be forgotten? What are the arguments your supporters have a tough time answering? What works in flipping supporters of your adversary over to your side? You will never get those answers from a single person completing a survey on their own, no matter how many single people you are asking.
You should continue to use traditional focus groups/mock trials when you want to benefit from reactions to a more complete presentation of the case, and when you want to witness the dynamics of the group processing their reactions to the case.
This is a situation where “useful” is not the same thing as “statistically predictive.” Traditional research won’t predict your in-court results, but will give you a way of working with likely reactions to your case. Large-scale research will give you a better handle on representative reactions, but will necessarily focus on more of a thumbnail version of the dispute. Ultimately, it is not a matter of a clear preference for either, but a matter of accepting the costs and benefits from each, and knowing that your purposes will point to a role for both.
____________________
Other Posts on Mock Trials:
- Trust Your Mock Trial (to a Point)
- Appreciate the Advantages of Online Mock Trials
- Beware of ‘Participation Deception’ in Your Surveys and Mock Trials
____________________