5 reasons the Senate race polls in Alabama are wrong

Who’s going to win the special election for the next U.S. Senator from Alabama? According to the two most recent polls, it’s either Republican Roy Moore by 9 points or Democrat Doug Jones by 10 points. How is this much variation even possible?

Unless a pollster stands over the shoulder of each individual who votes and watches, polls are samples. Samples have biases. In the case of this race, where the Washington Post says Moore had a bad habit of cruising on teenagers and trying to have sex with them, the biases have run amok. In my history working with hundreds of surveys at Forrester Research, I saw many of these challenges in surveys we did. So let’s take a look at five reasons why the polls are going to be wrong.

1. Half the polls miss people without landlines

As Nate Silver points out in his excellent article on fivethirtyeight, automated robo-dialer polls aren’t supposed to call mobile phones. (You may find this surprising given the number of spam calls you get on your smartphone, but the spammers set up operations to evade regulation, while pollsters need to operate in a lawful and transparent way.)

People with land lines are more likely to be white, older, and Republican. Pollsters can correct for that, but the corrections have problems, too (see point 3). And sure enough, Emerson College robopolls consistently show the Republican Moore ahead; his latest lead is 9 points.

2. People lie to pollsters

Will everyone tell a pollster the truth? Not if there’s social desirability bias. For example, when Nielsen used to collect viewing data by collating diaries that people filled out, PBS had higher ratings. Once Nielsen switched to automated recording of viewing (“PeopleMeters”), PBS ratings dropped. People reported what they imagined themselves to be — PBS viewers — instead of what they actually were.

In Alabama, there are multiple ways this could distort polls.

Republicans who are disturbed by Roy Moore’s purported behavior and are going to vote against him (or not vote) may tell a pollster they are voting for him, as most of their Republican friends probably are.

People who are going to vote for Moore despite his behavior may tell a pollster they are not voting for him out of shame.

Wives of Moore voters may be too shy to tell a pollster on the phone that they’re going to vote differently from their husbands.

Notice that these effects don’t all go in the same direction. But in a scandal-laden race like this, they’re impossible to escape. It’s worth noting, as Silver does, that polls conducted by humans (as opposed to robots) show more people voting against Moore. Fox News has Jones ahead by 10 points.

3. Weighting distorts results

What do you do if your poll reaches fewer black people or young people than it should according to past election results? You give the few you have more weight. Weighting is a common strategy in poll results to correct non-response bias — that is, to make up for people who aren’t represented.

There are about 3 million voters in Alabama. That means, in a typical poll of 1,000 people, each response represents 3,000 people. But a young African American voter that the poll reaches might count for 15,000 people if she’s one of a very few voters of her kind. Magnifying the sentiment of respondents like this increases the margin of error — there’s no reason to believe the few respondents in groups like this are any more representative of their age and ethnic group than anybody else. Failing to weight their responses, though, means your poll is overrepresenting other groups.

Think this doesn’t matter? SurveyMonkey revealed that its poll results shifted by 19 points — from a 9-point win for Jones to a 10-point win for Moore — depending on what weighting it used.

The net effect of the weighting might mean that a poll that ostensibly has a margin of error of four points has an actual error of eight points or more. And at that point, the poll become useless.

4. Polls miss late swings

The caricature of (white) Alabama voters is that they just go into the voting booth and vote for the candidate with an R next to his name. And it’s certainly true that Alabama has not elected a Democratic Senator since 1997.

But there are more Republicans in Alabama open to voting Democratic (or at least not voting for Moore) than in any recent memory. These are swing voters. And in the last few days they have seen:

• Moore’s accuser showing his signature in her high school yearbook.
• Fox News claiming the signature is forged.
• Fox News retracting its claim, because apparently the accuser added words to her yearbook inscription, rather than forged it.
• The other Alabama Senator, Republican Richard Shelby, who won his most recent election with 84% of the vote, saying he can’t vote for Moore because of the accusations against him.
• Moore’s wife, Kayla, saying she’s not anti-Semitic; in fact, “one of our attorneys is a Jew“. (This could swing votes in either direction.)

To the extent that these late developments change the minds of swing voters, the polls will miss the shift. Even the most recent poll results include people who expressed their opinions last Thursday and Friday; a lot has happened since then.

5. People may have trouble voting

Alabama’s strict voter ID laws — and the need to get to a motor vehicle registry to get a driver’s license or similar ID — might deter some poor, black, or rural voters.

Is it hard to imagine that a few pro-Moore voters might attempt to block Jones voters from voting?

Is it possible that violence will break out? Or that people waiting in line won’t get the chance to vote? Or that vote counts might be inaccurate?

I think this is going to be a close election (although, given the state of the polls, that has to be classified as a guess based on very little evidence). If so, we may not know for a very long time who actually won. And even if someone is declared the winner, expect lawsuits about the result.

Polls are imperfect. Especially in this race. Let’s just hope the election itself is less chaotic than the polls.