Error margins classically used to evaluate the precision of quota surveys underestimate the precision of political polls. Moreover, by definition, they don’t take into account potential biases. They are thus useless, or even perhaps counter productive.

Error margin in quota surveys

With a randomly collected sample, the error margin can be easily calculated using a well-known formula. This formula allows one to calculate the confidence interval around an estimated percentage as a function of the sample size, the estimated percentage and the chosen level of confidence. The below table give the error margin for the usual 95% confidence level :

	Estimated percentage
Number of interviews	20% ou 80%	30% ou 70%	40% ou 60%	50,0%
500	3,5%	4,0%	4,3%	4,4%
800	2,8%	3,2%	3,4%	3,5%
1000	2,5%	2,8%	3,0%	3,1%
1500	2,0%	2,3%	2,5%	2,5%
2000	1,8%	2,0%	2,1%	2,2%

 

Thus, a poll with 1000 respondents stating a 52% win for François Hollande on the eve of the 2012 presidential election would, in theory, correspond to a confidence interval of 52%-3,1%=48,9%/52%+3,1%=55,1%. Therefore, 50% is in the confidence interval, with a 95% chance, and the poll would not bring any relevant information.

A first snag is that there is no such thing as a random sample, in the same way as there is no Loch Ness monster. National statistical offices themselves actually don’t really do random sampling.

Small snag actually: it can be shown that the confidence interval coming from quota sampling can be well approximated by calculation similar to the one above (cf the paper by Jean-Claude Deville).

Using random sampling error margins as a proxy for quota sampling is thus quite acceptable. But political polls have more complex properties, that make those error margins calculation irrelevant for that particular case.

 

Random sampling error margins are not appropriate for electoral polls

Let us have a look at the polls published between the two rounds of the 2012 French presidential election. There were 21 of them (actually a bit more, because Ifop was publishing 3 days rolling polls. In order to avoid a digression on intersecting samples, the below table only shows some of the polls from Ifop). The score that is displayed is Francois Hollande’s, who was finally elected:

Before discussing the main point, let us note two things, to which we will come back:

–       All polls were pointing to the election of François Hollande, with a larger margin than the actual one (51,6%).

–       There is a clear decreasing trend. The average score of the lucky winner in all 21 polls is 53,7%. The average calculated on the last poll published by each agency is 52,8%

Let us discuss now the error margin and focus on the 8 polls published between May 2 and 4. These are the last polls published by each agency:

Ifop	Opinion Way	BVA	Ipsos	CSA	Harris Interactive	LH2	TNS
52%	52,5%	52,5%	52,5%	53%	53%	53%	53,5%

 

The maximum spread across these polls 1,5% (Ifop vs TNS). This is the crux of the matter: if political polls obeyed to the above mentioned error calculation, the spread across agencies should be larger.

We have simulated 1000 groups of 8 polls, with the sample sizes used by each agency, with a 52,8%probability (the average of the last 8 ones). Across these 1000 simulations, only 6 were showing a spread of 1,5% or less. In other words, the polls’ configuration on the eve of the second round of the 2012 presidential election, is an event of probability 0,6%.

Similar finding for the polls on the eve of the first round: the configuration of results for Nicolas Sarkozy and François Hollande had a 7% chance of happening.

And again for the score of Nicolas Sarkozy on the eve of the second round of the 20007 election. The spread across the 6 agencies was 2%: a 5% probability event.

When an event of such a small probability happens, statisticians conclude that the hypothesis supporting this probability calculation are wrong (random sampling error margins actually are based on this reasoning).

Polls of different agencies, which are different estimates of the same quantity, are less spread than error margin theory would allow them to be. They are thus more precise (see article on bias and precision) and their error margin is over estimated by the standard formula.

A conjecture on the precision of electoral poll

Let us brush aside the conspiracy explanation that would have research agencies working together to publish similar results.

A much more likely explanation of the larger than expected precision of political polls could be the following: the published results are calculated from collected data, with a weighting adjustment. This adjustment is based on the comparison between what interviewees say they voted at the last elections and the actual results of these. Before weighting, for example, the National Front (a far right group) scores are severely under estimated.

This weighting adjustment, most probably, is a stabilisation force. Sample theory tells us that the objective of post data collection weighting is to increase the precision of estimates. There is something similar happening here.

Something a little bit more complex, actually. Pollsters who choose the final weighting have of course in mind the collected data, but also their intimate knowledge of electoral sociology and all the other information they have at the time of the poll.

This added information, one fully quantitative – data about the votes of respondents at previous elections -, the other not easily translated in hard currency, makes political polls more precise than what error margin theory would tell.

By the way, there is nothing really unique, or illicit, with this idea of mixing quantitative and a priori non quantitative information: Nate Silver has some very convincing pages on how qualitative information can improve a quantitative model.

More precise but sometimes biased

The idea that political polls are more precise than what we commonly think probably doesn’t fit with the public perception…. This is due to the confusion between two very different concepts: bias and precision.

Let us have a look at the results of the first round of the French 2012 presidential election:

 

Candidate	First round result	Average of last polls	Error margin
Nathalie Arthaud	0,6%	0,4%	+/-0,4%
Philippe Poutou	1,2%	1,4%	+/-0,7%
Jean-Luc Mélenchon	11,1%	13,3%	+/-2,1%
François Hollande	28,6%	28%	+/-2,8%
Eva Joly	2,3%	2,4%	+/-1,0%
François Bayrou	9,1%	10,1%	+/-1,9%
Nicolas Sarkozy	27,2%	26,5%	+/-2,7%
Nicolas Dupont-Aignan	1,8%	1,7%	+/-0,8%
Marine Le Pen	17,9%	16,1%	+/-2,3%
Jacques Cheminade	0,3%	0,1%	+/-0,2%

The error margin is calculated under the assumption of a random sample of 1000 respondents, without any correction factor for small percentages (Nathalie Arthaud’s or Jacques Cheminade’s scores cannot be negative…). Let us keep in mind we believe this overestimatee the true error margin of an electoral poll.

Jean-Luc Mélenchon’s score is outside the confidence interval, as is most probably the one of Marine Le Pen. 2 candidates over 10 outside the confidence margin, this is far from the 95% confidence level. Most probably, the scores given by polls for these two candidates are biased.

Let us note this might not be the only bias. The average of the last polls on the eve of the second round were giving 52,8% to François Hollande, quite above the final 51,6%. For sure, the trend was downwards: polls are a snapshot at time t and some people might have changed their mind. Not impossible though that there was a slight overestimation: a positive bias. In that respect, it is interesting to point out that all left leaning candidates were globally overestimated at the first round, while those on the right were under estimated.

Possible reasons for this bias are discussed in the article on random sampling: non response – the fact that some people refuse to answer surveys – is the most likely reason.

 

Which error margin for a political poll?

As any survey, a political poll comes with two error risks:

–       A bias risk, due to the fact that some people will refuse to answer surveys,

–       A precision risk, due to the fact that only a sample is interviewed and not the whole population.

By definition, there is no such thing as an error margin against the first risk. If the bias were known, there would be no bias.

And, concerning the second risk, it is most likely that classical error margins under estimate the true error margins of electoral polls.

What is the point of publishing figures that are known to be false? This can only reinforce the defiance of citizens against data that are however useful and fuel the democratic debate.

Which way forward, then, if the general public is to be aware of error margins?

The answer probably is with the available back data of polls and electoral results. There are enough different agencies, and different elections, to set up an empirical database on polls precision. The matter of bias is more difficult, as probably dependent on the particular political situation, the personality of candidates and their positioning. It is anyway something that could be modelled.

 

Your turn

 Several other explanations could be given to explain this less than expected variance of results across different agencies. Here is a list of possible explanations. Which one do you prefer?

 

Related item

Bias

Endogeneity

Random sampling

 

References

J.C. Deville (1991). Une théorie des enquêtes par quota/ A theory of quota Survey – Survey Methodology/Techniques d’Enquête Vol 17 – n° 2, pp 163-181.

C. Gouriéroux (1981) : Théorie des sondages – Economica

N. Silver (2012): The signal and the noise; why so many predictions fail, but some don’t – The Penguin Press.