« PPBW - Hopkins Paper | Main | The Mixed Electoral System: Are rumors of its death greatly exaggerated? »

2 December 2005

Polls on polls

I came across an interesting article the other day which shows the results of a 1985 Roper Organization opinion poll which asked the public for their views on…opinion polls (full cite: Roper, Burns W. 1986. “Evaluating Polls With Poll Data�. Public Opinion Quarterly 50: 10-16). What do the people who take part in surveys think of them? A couple of the results stand out for their sheer humor value:

When asked about the type of person who takes part in a survey, a full 7% thought that pollsters interview “mostly unusual, nontypical, or even oddball types of people�. Unfortunately, the obvious follow-up question wasn’t asked, and we’re left to guess whether this 7% were projecting inside information about their own abnormality on to the rest of the sample.

When asked about the honesty of survey respondents, 16% believed that “only some� or “very few� people tell the truth to interviewers. Again, are these people projecting from their own dishonesty? I don’t know how we can begin to evaluate – let alone trust – the answers to this question if they were!

Beyond these questions, respondents were also asked how many times they had been interviewed in a poll prior to this one. The breakdown of the results was:

Never before - 41%
Once - 17%
Twice - 16%
3 to 5 times - 16%
6 or more times - 9%
Don’t Know - 1%

These numbers were far higher than I had expected, and raised a question in my mind.We know that taking part in a survey has an effect on respondents' attitudes: they become more interested in politics, and seem to learn more about current affairs after talking to an NES interviewer for an hour, for example. The worry here is that by virtue of taking part in a randomly sampled poll, a person is no longer representative of the public at large. We also know that there has been an explosion in the sheer number of opinion polls conducted since this Roper survey in 1985. The worry here is that many more people have previously taken polls than was the case in previous decades. At what point do we need to start controlling for a respondent’s prior survey participation when using polls to understand public attitudes?

Posted by Phil Jones at December 2, 2005 5:46 PM