Those Darn Polls

Rasmussen Reports released a new poll the other day, which found:

A new Rasmussen Reports national telephone survey finds that 48% of Likely U.S. Voters agree more with the Republican governor in his dispute with union workers. Thirty-eight percent (38%) agree more with the unionized public employees, while 14% are undecided.

The wording can be found here.

1* How closely have you followed news reports about the Wisconsin governor’s effort to limit collective bargaining rights for most state employees?


2* Does the average public employee in your state earn more than the average private sector worker in your state, less than the average private sector worker in your state or do they earn about the same amount?


3* Should teachers, firemen and policemen be allowed to go on strike?


4* In the dispute between the governor and the union workers, do you agree more with the governor or the union for teachers and other state employees?


5* Would you favor or oppose reducing your state government payroll 1% a year for 10 years, either by reducing the number of state employees or by cutting the pay of state workers?

As Nate Silver points out that it probably suffers from and order bias due to the question framing in prior questions.

The issue is clearest with the third question, which asked respondents whether “teachers, firemen and policemen” should be allowed to go on strike. By invoking the prospect of such strikes, which are illegal in many places (especially for the uniformed services) and which many people quite naturally object to, the poll could potentially engender a less sympathetic reaction toward the protesters in Wisconsin. It is widely recognized in the scholarship on the subject, and I have noted before, that earlier questions in a survey can bias the response to later ones by framing an issue in a particular way and by casting one side of the argument in a less favorable light.

The Rasmussen example is more blatant than most. While many teachers have been among the protesters at the State Capitol in Madison, obliging the city to close its schools for days, there have been no reports of reductions in police or fire services, and in fact, uniformed services are specifically exempted from the proposals that the teachers and other public-sector employees are protesting. So bringing in the uniformed services essentially makes No. 3 a talking point posed as a question.


As an analogy, imagine a survey that asked respondents whether they believed the Democrats’ health care overhaul included “death panels” before asking them whether they approved or disapproved of the bill over all.

Randomizing the order of questions can address this issue, or working harder to ask questions which use language free from this type of bias.The AFL-CIO had their own poll here.