By Dr. Ken Broda Bahm:
Imagine that you receive a phone call and the voice on the line offers participation in a research project, and also offers pay. Then, the caller walks you through a series of questions to determine your eligibility, and it becomes clear what the “right” answer is. And let’s say you could use the money. Do you shade your answers to lean toward what you think they’re looking for? Now, I like to think that readers of this blog, many of them lawyers, are more honest than the average person, so perhaps the answer is “probably not.” We know, however, that average citizens aren’t always truthful: as many as 80 percent will lie to their doctors about nutrition and exercise, for example. So, if they’re being recruited for a venue survey, or a mock trial or a focus group, will they lie about factors that should determine their eligibility?
The answer is that many of them will try. Recent research confirms that if compensation is at stake, then a portion of the potential participants will lie. In one of the first studies to focus specifically on the prevalence of deception to win eligibility, Philadelphia-area medical educators and practitioners (Lynch et al, 2019) asked about participants’ recent flu vaccination in a survey of 2275 American adults, soliciting research participants without offers of payment or with varying amounts of payment to see the conditions under which participants would lie in order to secure participation in the study. They found deception rates ranging from 10.5 percent to 22.8 percent. That suggests that deception should be a significant concern regarding both data quality as well as confidentiality when it comes to pretrial research. In this post, I will look at what can be done about it.
Guard Against Deception
When conducting research related to litigation, it is essential to screen participants. Naturally, you want people who are jury-qualified in the venue, and you also want to eliminate people who know too much, who are connected in some way to the parties or the law firms, or who have other characteristics that would make it unlikely that they would sit on the actual jury. When conducting that screening, you need to rely on the honesty of the answers you get. So how do you know it is honest? It’s never 100 percent, but there are several steps you can take to make your screening as robust as possible.
Don’t Signal the Right Answer. Taking the example used in the study, the question, “Have you had a flu shot in the last six months?” strongly implies that you are looking for people who have. To avoid signaling that, it is better to ask, “When, approximately, did you have your last flu shot,” and screen out individuals who answer that it was longer than six months ago or “never.” Questions that are open-ended or give more response options are going to be more reliable than a “Yes/No” question where the correct response is “Yes.”
Include Decoys. If you only ask about the parties involved in the case, then a potential participant might guess at what the case is about. That is why we will always include decoys in the screener. We will ask about the people and companies involved, but we will also include other names, including some very familiar ones like “Coca-Cola.” That not only keeps potential participants from guessing about the nature of the project, but it is also a way to flush out the dishonest. Someone who says they have never heard of the Coca-Cola company, for example, might not be telling the truth.
Ask More Than Once. We don’t just screen once when research participants are recruited by phone, we screen on-site as well. So if people are going to lie, they at least have to lie consistently, and as any prosecutor will attest, a surprising number of people are not able to do that. In addition, screening on-site is a good check against any unscrupulous market-research recruiter who signals the correct answer during the screening process in order to boost their recruiting quotas.
But Don’t Worry About Higher Pay
It is intuitive to think that a higher rate of pay would equal a higher risk of deception. After all, if more is at stake, then wouldn’t participants have a greater incentive to lie. But that is not what the researchers found. They tested payments varying from $5, $10, and $20 for a short study, and saw no significant relationship between the amount paid and the resulting deception. That top end compares to the $20 per hour that we generally use for compensation in our own research. That is important, because we have found that you need to pay a decent rate in order to get better commitment levels and greater representativeness in your recruited pool. If you pay only a minimum wage equivalent, then you will only get minimum-wage workers who will not be comparable to your actual jury.
With all that said, most potential mock jurors or other research participants are going to try to be honest, and I am happy to say that we have never had a known instance of participant dishonesty posing a threat to the confidentiality of a project. Still, you don’t know what you don’t know. So it pays to take every precaution.
Other Posts on Research Design:
- Be Design Conscious: Top Seven Posts on Mock Trial Design
- Avoid Market Research Mistakes in Your Mock Trial
- Follow 10 Simple Rules for Pretrial Research and Scientific Testimony
Lynch, H. F., Joffe, S., Thirumurthy, H., Xie, D., & Largent, E. A. (2019). Association Between Financial Incentives and Participant Deception About Study Eligibility. JAMA network open, 2(1), e187355-e187355.