A polling failure and a betting failure

Well, that went bad for the pollsters. Every poll published during the election campaign got it wrong. Collectively the polls suggested Labor would win around 51.5 per cent of the two-party preferred vote; at this stage in the count, it looks more like 49 per cent for Labor to the Coalition's 51 per cent.

I am as surprised as most. While it was obvious that the pollsters were doing something that reduced polling noise (and hopefully increased the polling signal), I assumed they knew what they were doing. What I really wanted was for the pollsters to tell us (the consumers of their information) how it was made: because it ain't what it says on the tin.

The 16 published polls since the commencement of the election campaign did not have the numerical features a statistician would expect from independent, representative and randomly sampled opinion polls. They did not look normally distributed around a population mean (even one that may have been moving over time). In short, the polls were under-dispersed.

I was troubled by the under-dispersion in the polls (here, here, and here), and I knew this could increase the risk of a polling failure. But I was not expecting a massive failure as such. Consistent with the polls, I thought the most likely outcome was a Labor victory in the order of 80 seats (plus or minus a few), with the Coalition to pick up around 65 and for others to land around 6 seats (80-65-6). The final result could end up being closer to 68-77-6. While a polling failure was possible, perhaps even 30 per cent likely, I did not think it the most likely outcome. Let's chalk it up to living in the Canberra bubble and confirmation bias.

I was also a little annoyed. The Bayesian aggregation technique I use makes the most use of the data at either end of the normal distribution around the population mean. Yet this data was implausibly missing on the public record. You don't need an aggregator when every poll result is in the range 48-49 to 51-52. There is nothing needing clarity on those results.

Because I assumed the pollsters were smoothing their own polls, I wondered what raw results they were actually seeing. Compared with February and March (Coalition on 47 per cent in round terms), the collective April and May poll results were substantially different (48.5 per cent). It is almost as if the public's mood shifted one and a half percentage points overnight with the 2 April Morrison Budget (and I am a long-standing sceptic about the capacity for Budgets to shift public opinion). To smooth so quickly to a substantially different number seemed unusual and analytically complicated. I wondered a number of times whether the pollsters had seen a 50 or a 51 or even a 52 for the Coalition in their raw data before smoothing (indeed, thinking about the missing inliers and outliers was how I got to being troubled by the polls).

What next: Something has to change. Like the United Kingdom, which had a similar scale polling failure with its 2015 general election, we need an inquiry into what went wrong. We also need way more transparency. Pollsters need to explain their methodology better and publish more on the pre-publication processing they undertake.

At least the myth of bookmakers knowing best has been put to bed. The bookmakers had a bad day too: especially Sportsbet, which had paid out early on a Labor win.

Postscript

Thanks to the Poll Bludger for the recognition. And some further reflections at Poll Bludger.

It is nice to see that my questioning of the under-dispersed in the polls means that I am now labelled a hardcore psephologist (albeit before the election).

The postmortem at freerangestats.info is worth reading.

A great election postmortem by Kevin Bonham.

The mathematics does not lie: why polling got the Australian election wrong, By Brian Schmidt.