By Dr. Ken Broda Bahm:
Legal persuaders who want to know their venue are interested in polls. They keep a finger on the national pulse, and especially on the way that pulse is registering at the local level. In addition to helping you know your potential jurors, that data on public opinion can also serve as a basis for expert witness testimony, and can matter in a change-of-venue motion. Sometimes, it can be used to assess the specific attitudes that will matter the most to the jury selection and persuasion in your own specific case. Recently, however, doubts about the accuracy and reliability of polling have peaked. Following the 2016 election, results that seemed to have surprised everyone, including the pollsters, it has become common to lament the “death of data,” and to observe that nearly every cause and candidate seems to have a poll showing itself as the most favored. That prompts the question for legal persuaders: Is it safe to rely on polling data?
A recent look at the accuracy of political polling around the world suggests that the answer is still a cautious “yes.” Research coming from the University of Houston tested models based on global electoral polling data, and show that data correctly picks more than 90 percent of election outcomes around the world. The study published in the journal Science also looked at other factors, like incumbency and the economy, and while we can still say “It’s the economy, stupid,” the analysis found that the best predictor was still the polling (Kennedy, Wojcik & Lazer, 2017). Of course, a finding like that is only surprising in a context of reduced faith in public opinion research. As the study’s lead author, Ryan Kennedy, notes, “It would be a mistake to abandon the enterprise. The future really is in trying to make better quantitative predictions.” Modern polling faces modern challenges, but don’t count the poll out just yet. In this post, I’ll take a look at both the reasons for distrust and the reasons for confidence, and what both say to the legal persuader.
The Trump Effect: Why We Think Polls Don’t Work
Belief in the demise of polling is driven not just by the surprising 2016 election results, but also by a misunderstanding of probability. For example, just before that election, Nate Silver’s influential polling site, FiveThirtyEight, predicted a 71.4 percent chance of Hillary Clinton winning. The morning of November 8th was widely seen as a repudiation of the site’s ‘Gold Standard’ status in the polling world, but a closer look indicates that we should not be so fast in seeing those results as “wrong.” A seven in ten chance of a Clinton win still means that there was about a three in ten chance of a Trump win — less likely, but still a substantial possibility. Just like a day with a 60 percent chance of rain can end up being sunny, sometimes the less likely result actually happens. As Ryan Kennedy notes, “We often think anything over 50 percent means absolutely an outcome is going to happen. That’s not necessarily the case.” So part of the solution is treating polling accurately, not as an iron-clad prediction, but as a statement about probability.
The Real Challenges
That said, making reliable uses of polling still means understanding and dealing with some real obstacles. And first among these is the need to account for the motivations behind the organization conducting the poll as well as the sampling methods and the questions themselves. Most pollsters are legitimate research and journalistic organizations, but some have a conscious or unconscious ideological bias, and if the pollster wants to make a given result more likely, there are myriad ways to push that result. Another problem has to do with the changing sample, and specifically the vanishing American landline telephone, which makes it essential to include cellphone numbers in any telephone survey. And whether the numbers are mobile or residential, people are increasingly comfortable with not answering or hanging up. That means that pollsters need to be persistent, and need to be dogged about holding the sample to stratified quotas. So, as always, the more professional the organization is, the better the survey data is going to be.
And the Reasons for Cautious Optimism
The ability of polling to still serve as a generally reliable predictor depends on the continued work of the professional organizations, and the continued ability of consumers of polling data to exercise sound judgment in discerning between the professionals and the spin-doctors. For legal persuaders, that means relying on data that is least ideologically motivated in describing the venue (e.g., so far at least, the U.S. Census) and in using what is available and what can be responsibly developed in digging deeper to look for case-relevant attitudes and experiences. A mock trial, for example, does not usually have a sample size that is robust enough to fully represent the venue. However, by aggregating that data across many projects, you can see some meaningful patterns.
There will always be obstacles and imperfections that limit the ability to fully know one’s audience. But legal persuaders need to be motivated to find out as much as they can. And the current cautiously optimistic reality is that polling still plays a role in that.
Other Posts on Public Opinion Research:
- Take Your Trial Venue’s Temperature: Seven Uses for a Community Attitude Survey
- The Trumpening: Take Some Lessons from an Unexpected Election Result
- Use Current Events in Voir Dire
Kennedy, K.; Wojcik, S. and Lazer. D. (2017). Improving election prediction internationally. Science, 355 (6324):515 DOI: 10.1126/science.aal2887