The tower of polls, biases, weights, and corrections

No matter what poll-based content you read, you are not just reviewing what the data show. You are reading data, corrected for bias, weighted for lack of representation, scored for bias, aggregated, and then manipulated by a model. Each of these “corrections” is a human decision. These models are valuable, but they do not represent reality.

Let’s examine what happens to poll data between the time that the pollster collects it until it reaches your eyes in a model.

Polls, samples, and bias

Where does the sample in poll data come from?

Some of it comes from random-digit dialing. Such polls have an obvious bias — they reach only people who answer their phones. They may oversample people with strong opinions and undersample people who care less, or are just busy.

Some of it comes from online samples generated by online advertising. Such polls have an obvious bias: they reach only people who click on ads, and miss people who have ad blockers.

Some of it comes panels of existing voters. These voters are sometimes compensated, which can influence their participation.

Once a pollster collects a poll, they review the sample to make sure it is “representative.” For example, the percentage of women or Black people or Hispanic people in the poll should match the percentage of such people in the voting population. If the percentages don’t match, the pollster will “weight” the poll accordingly. For example, if there are half as many Black people in the poll as in the voting population, the pollster will count the data from Black people twice as much. Weighting can correct for lack of representation, but it also increases error; you are taking a small subsample and magnifying their impact on the poll.

There is another form of weighting in poll results. Pollsters will attempt to get a number of Democrats, Republicans, or independents to match the numbered of registered party members who voted in the last election. Or they may ask people who they voted for in the previous presidential election, and attempt to make sure that their sample is corrected for the right number of Biden or Trump voters in a region from 2020. But such corrections are fraught. Some voters may not have voted in the previous election. Some may incorrectly recall who they voted for. Some may have changed their minds since the previous election and lie to cover up a vote they no longer feel comfortable about. So weighting poll results to match the number of voters who voted for candidates in the previous election is subject to all sorts of inaccuracies.

Finally, there is the question of how polls are reported. Some polls are conducted by candidates and parties. They may not choose to publish such polls, if they make a candidate appear to be losing badly. Conversely, they may not choose to publish a poll that makes their candidate appear to have a big lead, because such a poll might reduce turnout by making it appear that it’s not worth it to vote. Even polls conducted by unbiased organizations may not appear if the polling organization thinks the poll is too far from the consensus of other polls and would make them look bad (this is called “herding”). Clearly such choices by pollsters bias the aggregate of polls by withholding some information and publishing other information in ways that don’t reflect actual poll results.

Aggregation weights and biases

Organizations like FiveThirtyEight, The New York Times, RealClearPolitics, and The Washington Post publish poll aggregations. Such aggregations can tell you, for example, which candidate is leading nationally or in a given state and by how much.

However, such aggregations are not simple averages of polls. They account for recency, weighting more recent polls more heavily than older polls. They may account for “pollster quality,” weighting polls more heavily if the organizations that conduct them have been shown to be accurate or free of bias in the past. And they may reflect whether the polls were of registered voters or likely voters.

More sophisticated models like Nate Silver’s also reflect non-polling information. For example, Silver computes bounces from events like the party conventions, reallocates third-party and undecided voters, and corrects for economic fundamentals like inflation and unemployment to compute the win probability in any state. He even projects how those economic fundamentals are likely to change between now and the election.

All of these aggregation factors are judgments. It’s clear that just treating all polls equally is not the best way to aggregate probabilities in a given state or nationally. But every decision about what to weight and how much to weight it is a judgment that might also be in error.

The factors that can’t be modeled

Changes in election laws can significantly affect elections. For example, Georgia officials have asked to hand-count all ballots (although that is still being litigated) and Nebraska legislators were considering changing the state’s rules to award all the electoral votes to the statewide winner, instead of one vote to each congressional district as the rules currently require (this change appears to be dead for now).

Get-out-the-vote efforts can improve the chances for one party or another. In this election, Democrats may have a better organization and more money to devote to such efforts, but none of that is accounted for in polls.

The polls cannot capture the effect of October surprises, for example, if one candidate or another says or does something outrageous or is accused of doing so. They also cannot model the effects of deepfakes and false information being passed along on social media sites, or of election interference by nations like Russia or Iran.

And of course, no one knows what the weather will be like on November 5, or whether ballots mailed on time will be delivered on time by the post office.

Humans make models. And humans will decide the election.

After reading all this, you can better understand the wide range probabilities projected by the various election modelers and aggregators. They all stand atop the pile of biased and weighted polls aggregated by biased and weighted models and take their best shot, but what you read is the result of a set of human opinions about how elections work that may or may not match what is actually happening.

The results are not probabilities, they are projections. And with get-out-the-vote differences and events that haven’t even happened yet, they cannot get the “answer.”

So listen, choose, and vote. Because this is apparently a close election (unless the modelers are wrong about that, too) and nobody really knows what will happen.