Much is written about political opinion polling, yet very little is understood about how difficult it can be to get right. And it is not getting any easier.
Debates about polling usually begin with methodology, yet the question we should ask first is why we conduct polls, and then how.
Political opinion polls remain, at their core, social tools. When independently conducted and impartially reported, they make a valuable contribution to a functioning society. We know what people think, and so too does the Government of the day. That can only be a good thing.
Of course polls do not work unless they have the support of the media. How else do polls arbitrate important public conversations? The role of the media is central, but in recent times media pressures have been reshaping how polls are conducted, and not in a good way.
Polling the traditional “knock-on-doors” way takes time and investment. In a world of always-on and instant news, the push is towards faster and more frequent polling. To satisfy demand, there has been a move towards telephone and online polling. But at what cost?
To answer that question, we need to consider how effectively each approach delivers a pure sample of the population.
To achieve a pure sample requires that everyone in the population is given an equal chance of being selected for interview. Pure samples are only possible in theory, so the ambition of pollsters is to get as close as we can to the ideal. The further our sample is from the ideal, the more we need to use statistical weighing to make our sample look and act like a pure sample.
With face-to-face interviewing, the weighting required is minimal because the sample delivered quite closely aligns with that of the population. More weighting is usually applied to telephone samples. Online samples need to be heavily weighted.
More weighting does not automatically mean less accuracy. What it does is introduce another form of error – analyst error.
Weighting is a statistical technique. Every analyst will weight a sample differently. Some weighting strategies are not as effective as others which opens the door to polls getting it not just wrong, but badly wrong.
Weighting is not just statistically challenging for pollsters. It also presents an existential threat to polling. At what extreme of weighting or statistical interference do polls stop being measures of public opinion and start being models of public opinion? But that is an argument for another day.
Snapshot
Another albeit more subtle pressure being placed on polling by the media is the pressure to predict an election outcome. Polling can only provide a snapshot in time. Still, the temptation is to use every poll to predict no matter how distant the election at the time of interviewing. Whether it is because pollsters enjoy the publicity or because they feel resistance is futile, they have embraced the challenge of prediction rather than resist it.
In the most recent US election, pollsters and the media unwisely elevated the predictive power of polling to a new level by assigning probabilities to a particular outcome (85 per cent probability that Hillary Clinton would win, for example), giving rise to false perceptions of accuracy. Had polling organisations resisted turning percentages into probabilities, arguably, more commentators would have noticed that it was only the predictions of pollsters that were wrong, not their polling. You will no doubt see pollsters withdraw from the probability game in future elections.
Balancing the appetite for more polls with the need for more accurate polls will be a struggle. Science and experience suggest good statistics do not always compensate for poor sampling.
Ipsos MRBI and The Irish Times will continue with our programme of regular face-to-face polls while experimenting with new approaches and strategies. Our ambition is to be always on and always accurate, and if not, to be always accurate.
Damian Loscher is the managing deirector of Ipsos MRBI
