The people of Scotland are voting today in a referendum to decide whether Scotland should become an independent country (“Yes”) or remain in the United Kingdom (“No”), ending the 300 year old Treaty of Union. As shown by the poll tracker above (from the website whatscotlandthinks.org) opinion polls have consistently put the No vote ahead of Yes. In recent weeks the gap has narrowed but only two polls have put the Yes vote ahead. It looks like a narrow win for No, but can the polls be trusted?
Independence campaigner Craig Murray recently wrote on his blog.
I genuinely find it impossible to understand the gap between the opinion polls and what my eyes and ears tell me.
Mr Murray is, of course, entitled to be optimistic in the face of opinion polls that point to a victory for the other side. However, there are good reasons for thinking that there are some important structural biases in the opinion polls and that the final result of the referendum will not be as close as it seems.
Recently the Radio 4 programme More or Less addressed the question of poll accuracy. You can still listen to the recorded broadcast, which will be available for another year. I shall summarize the three main points:
- The opinion polls are not random survey samples, but are observational data from Internet panels. These panels consist of people who have enrolled to participate in periodic online surveys, mainly for market research purposes, in return for small financial rewards. As far as I know, only one classical telephone survey has been conducted by ICM Research (and this poll gives a narrow lead of 51% Yes to 49% No, well within the 3% nominal margin of error). The rest rely on the small, self-selected sub-population who participate in online panels and may therefore all be biased in the same way.
- In principle, the results of online panels can be extrapolated to the voting population by adjusting for demographic variables, such as age, sex, social class, … However, nobody knows what the voting population in the referendum will look like because this is an unprecedented event. Turnout is expected to be over 80%, much higher than in general elections, and will thus include people who do not normally vote. This, incidentally, is the theory favoured by Craig Murray.
- Voters may be shy about admitting they intend to vote No, preferring instead to say they are Undecided, or even that they will vote Yes. Martin Boon of ICM research calls this the “patriotic spiral of silence”. If true, this means the size of the No vote has been consistently underestimated by opinion polls. Some support for this comes from an analysis of previous constitutional referenda by Stephen Fisher, who shows that opinion polls have usually (but not always) exaggerrated the desire for constitutional change.
Putting these three issues together and you can see that the main sources of uncertainty in the opinion polls are very difficult to quantify but potentially quite large. We will find out what Scotland really thinks tomorrow.