A Covid poll with some very different findings to *that* viral Ipsos one

Earlier this week we wrote a post about an Ipsos MORI poll on Covid restrictions that had been commissioned by and written up in The Economist. The poll had gone viral because of some rather surprising and alarming findings, such as the idea that 19 per cent of Brits would support a permanent 10pm curfew, and 26 per cent would ban nightclubs and casinos.

We suspected the poll wasn’t particularly representative of how people really felt, and suggested that if the questions had been asked differently, and if other questions were asked, we might have seen a quite different set of results.

Well we no longer have to speculate on this, because a 70-person Oxford-based polling start-up called Prolific was inspired by our article and decided to carry out their own 24-hour survey, using a representative sample of 978 people in the UK — very similar to the Ipsos poll which surveyed 1,025 people — to see what the results would be if they changed the questions a little. As they explained to us:

We ran pretty much the same study as Ipsos, but we took into account the FT suggestions and adapted our survey accordingly. Specifically: 

We added a timeframe to the study, ‘Until Restrictions lift on July 19th’ We removed the word ‘rule’ from the study and replaced it with ‘idea’ 

We modified the wording of the scale items to make it seem less like a rule, e.g., ‘Having to wear masks in shops + on public transport’ → ‘Wearing a mask in shops and on public transport’ 

Prolific also changed the question that Ipsos had asked about restrictions remaining in place “permanently, regardless of the risk of Covid 19” to “permanently, even if there were little to no risk of Covid 19” — as we pointed out in our post earlier this week, the word “risk” tends to have negative associations,

And it turns out the results from all these changes were rather different. Specifically:

Just 3 per cent support a permanent curfew, compared to 19 per cent according to Ipsos.

Just 6 per cent support permanent closure of nightclubs, compared to 26 per cent according to Ipsos.

13 per cent supported a permanent 10-day quarantine when returning from foreign holidays, compared with 31 per cent according to Ipsos.

When it came to masks, there was slightly less of a difference between the two polls: 31 per cent said they should continue to be worn in shops and on public transport, compared with 40 per cent in the Ipsos poll who said wearing a mask in a public place should be mandatory.

Here is what the Prolific poll results look like as a whole, compared with the Ipsos results represented as crosses (full key below):

We asked Ipsos MORI chief executive Ben Page what he made of the Prolific poll, and he told us:

They changed the wording – to be honest I am not surprised because of that. By stressing less/no risk rather than “regardless of the state of the pandemic” you get different results – it’s not a high salience issue and question wording will therefore make a big difference . . . 

We can show you that simply adding four words to a statement about house building for example can make support go from 30% to 60%. It’s one reason why we often split sample questions to assess the impact of question wording on responses. 

This is exactly what we had suspected in the Ipsos/Economist poll: that the wording of the questions had made a big difference to people’s responses. But Page’s point about salience is very interesting and is not something we had considered: if an issue is not high up in people’s worry-list it seems that the wording of a poll’s questions has a bigger impact. This makes sense if you imagine that this might be the first time a respondent has even considered the question being asked of them, for example whether nightclubs and casinos should be closed. And this also explains why there was less of a difference in the two polls’ findings when it came to masks, which is a more politically sensitive and widely debated and therefore salient issue.

Page again:

It’s well worn in research circles. The key point is that low salience issues will tend to see bigger impacts of question wording and framing. Nightclubs are not top of mind for most people. 

He added that “the key thing is to look across a range of surveys — and indeed to look at qualitative and ethnographic work — to triangulate anything interesting.”

We would agree. It seems to us there is an important lesson here about the need to poke around at quantitative research just as much as you would qualitative findings; just because something has been given a number doesn’t make it any more true than anything else. Next time you see a poll or survey or any other data set or indeed news story that looks too implausible to be true, bear in mind that it might very well be.

Related links
Something’s up with this Covid poll (update) – FT Alphaville
Some Britons crave permanent pandemic lockdown – The Economist