Wednesday, October 22, 2008

Polls Are Smarter Than You   posted by ben g @ 10/22/2008 10:30:00 AM

Andrew Sullivan points to a post by DJ Drummond which makes the claim that the polls are significantly biased towards the Democrats. This is a perfect example of partisanship taking precedence over facts, and it thus deserves a thorough fisking. Drummond begins:

it needs noting that all of the major polling organizations are based in locations where liberals are strongest and conservatives weakest, where 'democrat' and 'republican' take on meanings wildly different from the rest of the country. The people making the executive decisions at these polls, most likely including the wording and order of polling questions, whether to focus on urban or suburban areas, the weighting of political affiliation, and the definition of 'likely voter', are most likely in regular contact and association with the most liberal factions of politics. It does not mean that they have deliberately skewed their decisions to support Obama, but it is obvious that there is an apparent conflict of interest in their process modality.

To begin with, most major *conservative* media outlets (e.g Fox News, the National Review) are located in these regions. Drummond suggests that through some kind of opinion osmosis, pollsters in urban areas tilt their polls towards liberals. The evidence suggests this contention is wrong:
  1. Fox News and Rasmussen (both Republican-owned) are going in the same direction as the average of all the other polls (including those funded by liberal/democrat organizations). The latest Fox Poll shows Obama leading by 9. Contrast this with the current pollster average difference of 8.
  2. Pollsters polled the Bush-Kerry election correctly, accurately predicting a close victory by Bush. Also, they under-polled Gore by 3 points.
  3. John McCain is campaigning most heavily in the swing states identified by the supposedly liberal-biased polls. His campaign, like all presidential campaigns, has its own internal polling. Apparently it matches up with Gallup et. al to a great degree.

Drummond goes on to make some specific arguments for his point of view:

most people do not have the interest to stop and take an 8-to-10 minute interview, especially from someone they do not know calling them up when they are likely to be busy doing something else. It's been established as well, that democrats in recent years are more willing to take part in polls than republicans, possibly due to perceived bias on the part of the media. But it is quite important to know if the pollsters were getting one person in ten to take the poll, or only one person in fifty, because the people not interviewed matter just as much as those who do participate. Yet I have never yet seen a poll this year that publishes response rates.

Although this has nothing to do with the claimed liberal bias of polling organizations, it is a generic methodological issue worth discussing. Response rates are typically in the 10 to 25% range (PDF link). In general, the candidate who excites their base more (regardless of party) will have a higher response rate from telephone polls. The argument is that this inflates estimates of their support. However, the evidence suggests the opposite; the candidate with more enthusiasm behind him/her inspires people not only to pickup the phone but also go to the voting booth.

It's also worth noting that Drummond ignores any methodological biases which would inflate McCain's estimated share of the vote. For example, people who either have no home phone or use cell phones are typically not being sampled. These people are disproportionately young and/or poor (both demographics favor Obama).

This is a big one that a lot of folks miss. I have noticed in the details, that all of the polls are asking about the public's opinion of the economy, and of their opinion of President Bush, even though he is not running this time.

These questions have been asked for decades, under both Republican and Democratic presidents. That Drummond is unaware of this shows that he doesn't know enough about polling to criticize it.

many polls ask a question about John McCain just after asking about the voter's opinion of President Bush, subtly linking the two men.

Polls that ask about the election first match the current polling trend. See, for example, that recent Fox News Poll where Obama leads by 9 (PDF link). This poll is in following with the majority of them in asking about Bush *after* they ask about the election.

no questions have been asked about approval of the specific performance of either Majority Leader Reid or Speaker Pelosi, and no other politician is linked to Barack Obama in the same way that polls link President Bush to John McCain.

First off, it's a rarity to poll about the performance of the Speaker or Majority Leader, regardless of whether they're Republican or Democrat. Second, it's not true that there haven't been polls dealing with Obama's connections. There have been several polls asking about voters' opinions on Obama's connections to Ayers and Wright.

Polls taken since Labor day have not mentioned foreign policy at all. There are no questions regarding Russia's invasion of Georgia, nor of Iran's nuclear weapons programs, nor about China's intentions viz a viz Taiwan, even though these are current events which have great significance in a presidential race, yet all of the polls are ignoring them. Again, the economy-only focus betrays a bias which violates the principles of the NCPP.

There have been polls on foreign policy since labor day, and it takes only a simple Google search to know this. See for example this one by the New York Times. The focus on the economy (both by the polls and by the candidates themselves) has not come about because of liberal bias, but because voters indicate that this is what matters to them.

The thing most folks forget about polls which get published in the media, is that the polls' first need is not to accurately reflect the election progress and report on actual support levels; it's about business.

This is a false dichotomy. These needs overlap to a great deal. SurveyUSA went from a no-name to the most respected pollster among bloggers (and, eventually the press) during the dem primaries simply because it more accurately predicted outcomes in the Democratic primary than anyone else. (They were frequently the outlier from the pack, by the way, and we'll get to that in a second). As a result they got more web traffic and citations. So there's definitely an economic and social incentive to give accurate polls. For a "conservative", this guy sure has a great disrespect for market efficiency.

you really think republicans or independents got more excited about Obama because of his convention, or that democrats and independents were more likely to vote for McCain because of the GOP convention? When you think about it, it should be obvious that these bumps are artificial unless there is a clear cause to show a change in support.

A simpler hypothesis is that the polling companies are accurately registering a slight increase in support for a given candidate in response to their increased positive media attention.

There has been unprecedented manipulation of demographics, corrupting even the raw data to the point where effective resolution of public opinion is doubtful. This might be described as an honest mistake, if one is willing to accept greed as an honest motive. Gallup, for example, who has more experience than any other polling group and who therefore should have known better more than anyone else to fiddle with the weights.

Where is the data? Where is the evidence that Gallup is not weighting demographics accurately? Drummond says he has written on it previously, but a search of his site for "gallup census" shows no posts which actually show the gallup weighting to be at odds with the US census. Demographics weighting varies as a function of the pollster. It's worth noting that despite their different weightings, the major pollsters agree that Obama is leading by at least 5 points right now.

So OK, Gallup is having a bad year, but what about the rest? Well, there the phrase to consider is follow the leader.

I've been following the polls since the primaries, and I can safely say: that hasn't been very true for this election or the dem primaries. There have been spreads as great as 15 points (see for example New Hampshire in the Democratic Primary) between various pollsters at several points in this year and last. The models, and occasionally the outcomes, have been significantly different from one pollster to the next. This guy needs to compare SurveyUSA to Gallup to Public Policy Polling before he writes another post accusing them of following one another.

So, could I be wrong? I have to be honest and admit that I could.

That's good to hear, because he actually is wrong. Here's why:

That McCain is more experienced with the key issues than Obama was ignored, that the historical significance of the debates shows that the effects appear several weeks later was also ignored. That the economy could be as reasonably blamed on the democrat-controlled Congress as on the republican President was never considered. That character would be a salient factor in the decisions of voters was rejected out of hand.

Wishful partisan thinking. Drummond wishes that people supported his president and candidate and issues, but because they don't he criticizes the data which proves otherwise.