By Dr. Ken Broda Bahm:
In a recent piece in Forbes online, Roger Dooley tells his own story of a spectacular market research failure. His company was involved in metal sales and competed in a market with an overall price similarity. Thinking that their product had some quality advantages over the competition, the company wanted to charge a higher price. The market research they did showed that price came in dead last as a concern, with likely customers focused on quality, delivery speed, and customer service instead. With that reassuring message in hand, up went the price…and down went the sales. It turns out that price mattered after all, and it mattered a great deal. So why didn’t the market research catch that? Because there were at least two problems with the data they collected. One, they looked at attitudes and not behavior. People like to think that they’re not so stingy as to put price above quality, but in this case they did. Two, they failed to account for changed circumstances. Sure, price didn’t matter when all the suppliers were essentially charging the same price. But when one supplier started charging more, then suddenly price matters.
Roger Dooley’s audience for this piece is market researchers, but the critique and the caution applies just as well to those who conduct litigation focus group and mock trial projects. There are definite similarities between the legal pretrial research and the market research fields, since we often use the same recruiters, the same facilities, and have similar goals in testing the public’s reaction to new information. Some litigation consultants even have a background in market research. So for this post, I’m going to draw some lessons from this market researcher’s observations and share some thoughts on what not to do in your legal small group research.
In seeking to avoid bad habits that may have come from market research or other settings, I think four pieces of advice stand out.
1. Separate Beliefs, Beliefs About Behavior, and Behavior
Treat these as three different things: First, there is what you believe; second, there is what you believe about what you would do in the future; and third, there is what you would actually do. A fundamental mistake in any kind of research is to blur these distinctions. As Dooley writes in the piece, “People are often incapable of articulating why they do things or how they would behave in the future.” As we have also noted, the research shows a wide difference between the articulated rationale people give and the underlying motivator for it. So it helps to think of those three levels. Sometimes beliefs matter because beliefs are what jurors will be sharing in deliberation. Sometimes predictions of future behavior matter, because jurors are likely to apply those same predictions to other people. But most often, behavior matters: What will jurors actually do in response to new information?
2. Vary the Circumstances
By nature, a focus group or a mock trial exposes a mock jury pool to limited information. You cannot recreate the full fact pattern and you cannot duplicate the ebb and flow of an actual trial. Based on that reality, it is wise to remind yourself of the limits and the fact that you are testing a “thin slice” of the case. That thin slice can be enormously influential because it captures the reaction to the overall story, and that reaction will exert a strong pull on all the evaluations down the road. But the thin slice does not capture everything. For that reason, when you can, you should vary the circumstances that you are testing in your research. Give jurors more information in the final interview and see how they react to the revised landscape. Or even better, divide out the groups so two mock juries hear the basic story and two other mock juries hear that story with additional information (e.g., a key document or expert subject to a motion in limine).
3. Trust What You Test, More than What You Ask
Self-reported data, Dooley notes, “can be fine for simple facts like, ‘Did you eat breakfast today?'” but it will rarely answer questions like, ‘Why do you prefer Grey Goose vodka?'” Asking about attitudes or reasons for behaviors makes your research subjects or mock jurors into interpreters. Testing, on the other hand, makes you the interpreter. If you want to know whether jurors are primed to believe a large company is more likely than another party to have been dishonest, you could ask that question directly: “Would knowing that a given party in litigation is a large company influence your likelihood of finding that party had been dishonest?” They will answer, but that answer will be strongly influenced by the social desirability of fairness. Most will say the fact of a large company would not make a difference. But if you test that, for example by providing a one paragraph description of a scenario, varying whether the party is identified as an individual or a corporation, you are likely to get a different answer, one much less favorable to the corporation. This is parallel to Dooley’s recommendation to market researchers: If you want to see if a 20 percent discount makes a difference, don’t ask about it, test it.
4. Generally, Use Focus Groups to Measure Attitudes and Mock Trials to Measure Behavior
The dominant ways we test in a litigation context are through focus groups and mock trials. Now, a terminological note is in order, since these terms are sometimes used interchangeably. To us, a focus group describes a project in which a legal case is presented piece by piece (for example, statement of the case, then liability, then cause, then damages) interspersed with detailed interviews. A mock trial, on the other hand, is a project in which the goal is to more closely simulate the trial and to put primary emphasis on the jury’s unmediated deliberations. A focus group does a good job of unearthing at least the explicit attitudes jurors will bring to the case. For that reason, a focus group is helpful to conduct during discovery, so you can answer the questions, “What information is clear and what isn’t?” and “What questions do you have?” A mock trial, where you’re able to watch jurors deliberate on their own, is the only way to view the behavior of coming to a verdict. That is the point at which the “what they feel,” that is so often the focus of market research, becomes the “what they do” that litigators need to know.
When trying to elicit information from other humans, there is a sliding scale of reliability. On one end of the scale, there is the purely descriptive (“Do you work for a company with more than 500 employees?”), and on the other end of the scale, there is the completely subjective (“Do large companies help or hurt the American dream?”). The tools you choose will depend on where your interest lies on that spectrum. The more a decision carries an emotional component, the less researchers can depend on self-reports on the causes or reasons for that decision.
______
More Posts on Focus Group or Mock Trial Research:
- Don’t Be Entranced By Statistical Claims From MockTrial Research
- Don’t Ride with ‘Frequent Flyers’ in Your Mock Trial Research
- Know Your Constraints: A Conversation on Mock Trial Design
______
Image Credit: 123RF.com, Used under license.