webdiplomacy.net

Some of you may remember the Yes, Prime Minister episode in which it was argued that the wording and ordering of of opinion poll questions can be manipulated to achieve desired outcomes.

https://youtu.be/ahgjEjJkZks?si=Je0kDKbFAgF4NcLZ

Well, Ipsos have only gone and done it!

Results below

https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc548f657-0c77-45ae-baf6-2bcab076fe69_1220x1128.png?utm_source=substack&utm_medium=email

That's fantastic, highly amusing, well done to them.

I am a big fan of Yes, Minister and Yes, Prime Minister.

Yes, you can ask leading questions to get the desired outcome... in spite of public opinion. The sum of the matter is that you should take poll results with a grain of salt.

Enjoyable, very small difference though.

How big do you think the error bara are?

Can we compare the same question to just asking 5 unrelated questions in advance of the final ones?

Small difference? It went from a clear majority (45 to 38) in favour of national service to an even clearer majority (48 to 34) against. A 21 point swing is pretty significant in any context, and if error bars are big enough to hide it then it's pointless doing any polling at all. Elections are typically decided by somewhat smaller margins.

I feel this thread is really missing the other side of the story.

These issues with polling are well understood and sophisticated pollsters have fairly reliable methods to reduce these errors. For example, the question-order bias can be minimized by pre-testing the questions, randomizing / rotating the question order, inserting neutral / buffer questions, etc.

It's true that a survey can be cynically manipulated to produce the "intended" results. But it's also true that a serious pollster with a reputation to defend can't constantly release biased polls, especially when the poll's bias can be tested against real data (e.g., an election outcome).

So when an interest group releases a one-off poll about something unverifiable, it makes sense to be very skeptical. But when 538 does a careful meta-poll about some real world event (e.g., who is going to be the Governor of California), it makes sense to assume it has some predictive power.

Esquire Bertissimmo wrote: ↑

Mon Mar 04, 2024 8:13 pm

But it's also true that a serious pollster with a reputation to defend can't constantly release biased polls, especially when the poll's bias can be tested against real data (e.g., an election outcome).

A Machiavellian nitpick: Polls are usually made every few weeks in the run-up to the election, so only the last polls need to be accurate. You can always explain the apparent shift by campaigning.

learnedSloth wrote: ↑

Tue Mar 05, 2024 11:37 am

Esquire Bertissimmo wrote: ↑

Mon Mar 04, 2024 8:13 pm

But it's also true that a serious pollster with a reputation to defend can't constantly release biased polls, especially when the poll's bias can be tested against real data (e.g., an election outcome).

A Machiavellian nitpick: Polls are usually made every few weeks in the run-up to the election, so only the last polls need to be accurate. You can always explain the apparent shift by campaigning.

I'm still reading into this a desire to say polls are mostly phoney, which isn't necessarily the case.

A well done poll does actually reflect public opinion at the time it's conducted. It doesn't exist in a vacuum — it can be tested against other polls, other sources of opinion research (e.g., betting markets, focus groups, etc.), and strange results get huge public scrutiny, so there are strong incventives to not fabricate nor bias one's results. Polls should change in response to events, but in the absence of big events, professional pollsters prize poll stability as a sign of good methodology. There are good reasons to think that the polls that best predict the final outcome (e..g., big well-conducted meta-polls) were accurately tracking opinion along the way.

Betting shops have obvious monetary incentives to get their odds right. I think that even Machiavelli would have hard time sketching ways to skew them. It would definitely require deep pockets because of said incentives.

Octavious wrote: ↑

Fri Mar 01, 2024 4:54 pm

Small difference? It went from a clear majority (45 to 38) in favour of national service to an even clearer majority (48 to 34) against. A 21 point swing is pretty significant in any context, and if error bars are big enough to hide it then it's pointless doing any polling at all. Elections are typically decided by somewhat smaller margins.

I was misreading the result as a 3% shift, 45 -> 48 %, barely outside the realm of error bara.

Brilliant series. The elevation to Prime Minister episodes were fantastic