The Polls Have It Wrong

by CKR

Articles like this give me a bad feeling.

I’ve been thinking since the early primaries that there is something fundamentally wrong with the way the opinion polls are structured. They divide up the country into neat categories. Let’s take a Pew Research poll. Pew is an experienced and reliable organization within the polling community.

It starts out with divisions into Republicans and Democrats. That’s fair enough, since the poll, from before “Super Tuesday,” is about how people will vote.

But then it goes to the traditional “demographics:” Men, women, white, black, white men, white women, age groups, education, income, religion, primary date on which they will be voting.

These categories are easy to come by. Just ask the people you’re polling. But right away we can see problems: “white” and “black” hardly cover all the ways people think of themselves now, not to mention that it leaves out Latinos or sorts them into those two big baskets.

In the science I’ve done, the reason for breaking data down into categories is to try to pick up patterns; that’s what the polls try to do too. But in my science, I try to make the categories correspond to how cause and effect might work. That’s easier in chemistry than in polling. In chemistry there are some basic principles that limit cause and effect.

The polls are constantly interpreted as if they enlighten us as to cause and effect. From that article I linked up top:

In the fierce campaign between Sens. Hillary Rodham Clinton and Barack Obama, a battle dominated by questions of race and gender, white men have emerged as perhaps the single critical swing constituency.

Lots of assumptions in that single sentence.

An examination of exit polls in Wisconsin and Ohio, states with striking similarities, shows that many more working-class white men in Ohio said race was a factor in their vote on March 4 than was the case in Wisconsin. The analysis makes clear that race was not the deciding factor in the Ohio primary but did contribute to Clinton's margin of victory.

Even more assumptions here. In my science, I’d try to hold variables constant so that I could look at one at a time. This isn’t possible in polling, The best that can be done is encapsulated in “states with striking similarities.” But people in those two states are not identical, and the votes were on different days, with much news and conversation in between. That “single critical swing constituency” represents a choice on the part of the analyst. Depending on what one wants to focus on, the difference in any election can be attributed to many groups, or the difference can be found in a composite of groups.

The question of how groups are defined has been my main concern. The implied causation here is that being a working-class white man causes you to vote for Hillary Clinton. This is supported by the question in the exit polling and by the fact that Clinton garnered more votes than Obama did from this group. But that last is circular: we know that being a working-class white man causes you to vote for Hillary Clinton because more working-class white men voted for Hillary Clinton. And not every working-class white man voted for Hillary Clinton, so that is not a determinative variable. The choice of that question says something about the expectations of the polling organization, not necessarily about the mindset of the voters. Of course, “Did you vote against Obama because he is black?” would probably have obtained a bunch of lies unreliable results.

When I talk to friends, it seems to me that the common wisdom, most notably about women of a certain age, just isn’t working. What I am hearing suggests that use of the internet may have something to do with it. And that’s not a bad category: it would explain some of the age differences. If those age differences are in fact due to something other than years spent breathing, like hours spent in communication with people you’ve never met face to face via electronic devices, then this may be a better predictor of voting behavior. But it may be harder to deal with by the polling organizations.

We also see an influx of voters in these primaries. Why have those people stayed away in the past, or why have they decided to vote now? Those are different questions, and their answers may sort people differently. Perhaps these new (or returned) voters find the candidates inspirational. Perhaps they have learned, through the past eight years of contempt by the elected for the voters, that citizen participation is important. Arranging the questions and multiple-choice answers in various ways will give different nuances. “Fed up with Bush” and “Inspired by Clinton/Obama” might be useful categories.

The polling organizations would like to have similar categories for every election, which is part of the reason that “demographics” predominate. But because the polls drive the campaigns, and because the polls divide the electorate in particular ways, the campaigns move toward “demographic”-driven strategies (we need a few more working-class white men, and maybe some urban Latina women). If you aim your message at working-class white men and urban Latina women, you probably will get more of those groups to vote for your candidate. But does this say more about those groups or about the campaign tactics? Further, the “demographics” are not fully separable from racism, as we are now seeing.

Yes, it makes sense for Dan Balz to do his analysis that way; it most likely parallels what the candidates are doing. But it is also why the polls are unreliable. The degree of their unreliability this year suggests that their categories are becoming less and less indicative of what is going on in the voters’ minds.