Why I am troubled by the polls

A couple of days ago I made the following statement:

I must admit that the under-dispersion of the recent polls troubles me a little. If the polls were normally distributed, I would expect to see poll results outside of this one-point spread for each side. Because there is under-dispersion, I have wondered about the likelihood of a polling failure (in either direction). Has the under-dispersion come about randomly (unlikely but not impossible). Or is it an artefact of some process, such as online polling? Herding? Pollster self-censorship? Or some other process I have not identified?

Since then, we have had two more polls in the same one point range: 51/52 to 49/48 in Labor's favour. As I count it on the Wikipedia polling site, with a little bit of licence for the 6-7 April poll from Roy Morgan, there are thirteen polls in a row in the same range.

One of the probability exercises you encounter when learning statistics is the question: How likely is one to flip a coin thirteen times and throw thirteen heads in a row. The maths is not too hard. If we start with the probability of one head.

$$P(H1) = \frac{1}{2} $$

The probability of two heads in a row is

$$P(H2) = \frac{1}{2} * \frac{1}{2} = \frac{1}{4}$$

The probability of thirteen heads is

$$P(H13) = \biggl(\frac{1}{2}\biggr)^{13} = 0.0001220703125$$

So the probability of me throwing 13 heads in a row is a little higher than a one in ten thousand chance. Let's call that an improbable, but not an impossible event.

We can do something similar to see how likely it is for us to have 13 opinion polls in a row within a one-percentage-point range. (Well, actually two percentage points when you account for rounding). Let's, for the sake of the argument, assume that for the entire period the population-wide voting intention was 48.5 per cent for the Coalition. This is a generous assumption. Let's also assume that the polls each had a sample of 1000 voters, which implies a standard deviation of 1.58 percentage points.

$$SD = \frac{\sqrt{48.5*51.5}}{1000} = 1.580427157448264$$

From this, we can use python and cumulative probability distribution functions - the .cdf() method in the following code snippet - to calculate the probability of one poll (and thirteen polls in a row) being 48 or 49 per cent when the intention to vote Coalition across the whole population is 48.5 per cent.

import scipy.stats as ss
import numpy as np

sd = np.sqrt((48.5 * 51.5) / 1000)
print(sd)

pop_vote_intent = 48.5
p_1 = ss.norm(pop_vote_intent, sd).cdf(49.5) - ss.norm(pop_vote_intent, sd).cdf(47.5)
print('probability for one poll: {}'.format(p_1))

p_13 = pow(p_1, 13)
print('probability for thirteen polls in a row: {}'.format(p_13))

Which yields the following results

probability for one poll: 0.47309677092421326
probability for thirteen polls in a row: 5.947710065619661e-05

Counter-intuitively, if the population-wide voting intention is 48.5 per cent; and a pollster randomly samples 1000 voters, then the chance of the pollster publishing a result of 48 or 49 per cent is slightly less than half.

The probability of 13 polls in a row at 48 or 49 per cent is 0.000059. This is actually slightly less likely than throwing 14 heads in a row.

I get the same result if I run a simulation 100,000,000 times, where each time I draw a 1000 person sample from a population where 48.5 per cent of that population has a particular voting intention. In this simulation, I have rounded the results to the nearest whole percentage point (because that is what pollsters do).

Again we can see only 47.31 per cent of the samples would yield a population estimate of 48 or 49 per cent. More than a quarter of the poll estimates would be at 50 per cent or higher. More than a quarter of the poll estimates would be at 47 per cent or lower. The code snippet for this simulation follows.

import pandas as pd 
import numpy as np 

p = 48.5
q = 100 - p
sample_size = 1000
sd = np.sqrt((p * q) / sample_size)
n = 100_000_000
dist = (pd.Series(np.random.standard_normal(n)) * sd + p).round().value_counts().sort_index()
dist = dist / dist.sum()
print('Prob at 50% or greater', dist[dist.index >= 50.0].sum())

# - and plot results ...
ax = dist.plot.bar()
ax.set_title('Probability distribution of Samples of '+str(sample_size)+
    '\n from a population where p='+str(p)+'%')
ax.set_xlabel('Rounded Percent')
ax.set_ylabel('Probability') 
fig = ax.figure
fig.set_size_inches(8, 4)
fig.tight_layout(pad=1)
fig.text(0.99, 0.01, 'marktheballot.blogspot.com.au',
        ha='right', va='bottom', fontsize='x-small', 
        fontstyle='italic', color='#999999') 
fig.savefig('./Probabilities.png', dpi=125) 

As I see it, the latest set of opinion polls are fairly improbable. They look under-dispersed compared with what I would expect from the central limit theorem. My grandmother would have bought a lottery ticket if she encountered something this unlikely.

In my mind, this under-dispersion raises a question around the reliability of the current set of opinion polls. The critical question is whether this improbable streak of polls points to something systemic. If this streak is a random improbable event, then there are no problems. However, if this streak of polls is driven by something systemic, there may be a problem.

It also raises the question of transparency. If pollsters are using a panel for their surveys, they should tell us. If pollsters are smoothing their polls, or publishing a rolling average, they should tell us. Whatever they are doing to reduce noise in their published estimates, they should tell us.

I am not sure what is behind the narrow similarity of the most recent polls. I think pure chance is unlikely. I would like to think it is some sound mathematical practice (for example, using a panel, or some data analytics applied to a statistical estimate). But I cannot help wondering whether it reflects herding or pollster self-censorship. Or whether there is some other factor at work. I just don't know. And I remain troubled.

A systemic problem with the polls, depending on what it is, may point to a heightened possibility of an unexpected election result (in either direction).