Polls Are Smarter Than You

Share on FacebookShare on Google+Email this to someoneTweet about this on Twitter

Andrew Sullivan points to a post by DJ Drummond which makes the claim that the polls are significantly biased towards the Democrats. This is a perfect example of partisanship taking precedence over facts, and it thus deserves a thorough fisking. Drummond begins:

it needs noting that all of the major polling organizations are based in locations where liberals are strongest and conservatives weakest, where ‘democrat’ and ‘republican’ take on meanings wildly different from the rest of the country. The people making the executive decisions at these polls, most likely including the wording and order of polling questions, whether to focus on urban or suburban areas, the weighting of political affiliation, and the definition of ‘likely voter’, are most likely in regular contact and association with the most liberal factions of politics. It does not mean that they have deliberately skewed their decisions to support Obama, but it is obvious that there is an apparent conflict of interest in their process modality.

To begin with, most major *conservative* media outlets (e.g Fox News, the National Review) are located in these regions. Drummond suggests that through some kind of opinion osmosis, pollsters in urban areas tilt their polls towards liberals. The evidence suggests this contention is wrong:

  1. Fox News and Rasmussen (both Republican-owned) are going in the same direction as the average of all the other polls (including those funded by liberal/democrat organizations). The latest Fox Poll shows Obama leading by 9. Contrast this with the current pollster average difference of 8.
  2. Pollsters polled the Bush-Kerry election correctly, accurately predicting a close victory by Bush. Also, they under-polled Gore by 3 points.
  3. John McCain is campaigning most heavily in the swing states identified by the supposedly liberal-biased polls. His campaign, like all presidential campaigns, has its own internal polling. Apparently it matches up with Gallup et. al to a great degree.

Drummond goes on to make some specific arguments for his point of view:

most people do not have the interest to stop and take an 8-to-10 minute interview, especially from someone they do not know calling them up when they are likely to be busy doing something else. It’s been established as well, that democrats in recent years are more willing to take part in polls than republicans, possibly due to perceived bias on the part of the media. But it is quite important to know if the pollsters were getting one person in ten to take the poll, or only one person in fifty, because the people not interviewed matter just as much as those who do participate. Yet I have never yet seen a poll this year that publishes response rates.

Although this has nothing to do with the claimed liberal bias of polling organizations, it is a generic methodological issue worth discussing. Response rates are typically in the 10 to 25% range (PDF link). In general, the candidate who excites their base more (regardless of party) will have a higher response rate from telephone polls. The argument is that this inflates estimates of their support. However, the evidence suggests the opposite; the candidate with more enthusiasm behind him/her inspires people not only to pickup the phone but also go to the voting booth.

It’s also worth noting that Drummond ignores any methodological biases which would inflate McCain’s estimated share of the vote. For example, people who either have no home phone or use cell phones are typically not being sampled. These people are disproportionately young and/or poor (both demographics favor Obama).

This is a big one that a lot of folks miss. I have noticed in the details, that all of the polls are asking about the public’s opinion of the economy, and of their opinion of President Bush, even though he is not running this time.

These questions have been asked for decades, under both Republican and Democratic presidents. That Drummond is unaware of this shows that he doesn’t know enough about polling to criticize it.

many polls ask a question about John McCain just after asking about the voter’s opinion of President Bush, subtly linking the two men.

Polls that ask about the election first match the current polling trend. See, for example, that recent Fox News Poll where Obama leads by 9 (PDF link). This poll is in following with the majority of them in asking about Bush *after* they ask about the election.

no questions have been asked about approval of the specific performance of either Majority Leader Reid or Speaker Pelosi, and no other politician is linked to Barack Obama in the same way that polls link President Bush to John McCain.

First off, it’s a rarity to poll about the performance of the Speaker or Majority Leader, regardless of whether they’re Republican or Democrat. Second, it’s not true that there haven’t been polls dealing with Obama’s connections. There have been several polls asking about voters’ opinions on Obama’s connections to Ayers and Wright.

Polls taken since Labor day have not mentioned foreign policy at all. There are no questions regarding Russia’s invasion of Georgia, nor of Iran’s nuclear weapons programs, nor about China’s intentions viz a viz Taiwan, even though these are current events which have great significance in a presidential race, yet all of the polls are ignoring them. Again, the economy-only focus betrays a bias which violates the principles of the NCPP.

There have been polls on foreign policy since labor day, and it takes only a simple Google search to know this. See for example this one by the New York Times. The focus on the economy (both by the polls and by the candidates themselves) has not come about because of liberal bias, but because voters indicate that this is what matters to them.

The thing most folks forget about polls which get published in the media, is that the polls’ first need is not to accurately reflect the election progress and report on actual support levels; it’s about business.

This is a false dichotomy. These needs overlap to a great deal. SurveyUSA went from a no-name to the most respected pollster among bloggers (and, eventually the press) during the dem primaries simply because it more accurately predicted outcomes in the Democratic primary than anyone else. (They were frequently the outlier from the pack, by the way, and we’ll get to that in a second). As a result they got more web traffic and citations. So there’s definitely an economic and social incentive to give accurate polls. For a “conservative”, this guy sure has a great disrespect for market efficiency.

you really think republicans or independents got more excited about Obama because of his convention, or that democrats and independents were more likely to vote for McCain because of the GOP convention? When you think about it, it should be obvious that these bumps are artificial unless there is a clear cause to show a change in support.

A simpler hypothesis is that the polling companies are accurately registering a slight increase in support for a given candidate in response to their increased positive media attention.

There has been unprecedented manipulation of demographics, corrupting even the raw data to the point where effective resolution of public opinion is doubtful. This might be described as an honest mistake, if one is willing to accept greed as an honest motive. Gallup, for example, who has more experience than any other polling group and who therefore should have known better more than anyone else to fiddle with the weights.

Where is the data? Where is the evidence that Gallup is not weighting demographics accurately? Drummond says he has written on it previously, but a search of his site for “gallup census” shows no posts which actually show the gallup weighting to be at odds with the US census. Demographics weighting varies as a function of the pollster. It’s worth noting that despite their different weightings, the major pollsters agree that Obama is leading by at least 5 points right now.

So OK, Gallup is having a bad year, but what about the rest? Well, there the phrase to consider is follow the leader.

I’ve been following the polls since the primaries, and I can safely say: that hasn’t been very true for this election or the dem primaries. There have been spreads as great as 15 points (see for example New Hampshire in the Democratic Primary) between various pollsters at several points in this year and last. The models, and occasionally the outcomes, have been significantly different from one pollster to the next. This guy needs to compare SurveyUSA to Gallup to Public Policy Polling before he writes another post accusing them of following one another.

So, could I be wrong? I have to be honest and admit that I could.

That’s good to hear, because he actually is wrong. Here’s why:

That McCain is more experienced with the key issues than Obama was ignored, that the historical significance of the debates shows that the effects appear several weeks later was also ignored. That the economy could be as reasonably blamed on the democrat-controlled Congress as on the republican President was never considered. That character would be a salient factor in the decisions of voters was rejected out of hand.

Wishful partisan thinking. Drummond wishes that people supported his president and candidate and issues, but because they don’t he criticizes the data which proves otherwise.



  1. 1) does wizbang allow non-retards to post? 
    2) look at ruy teixeira in 2004 trying to explain why the polls shouldn’t be trusted (they favored bush, though more marginally than the ones which favor obama right now). sorry to engage in a game of equivalence…. 
    3) look at how far the polls were off in 2000 and 2004.

  2. It’d be fun to listen to this guy arguing with his oncologist. Look, there’s a right way to deal with a loss, and it isn’t retreating into a fantasy world:  
    “Thought shall be the harder, heart the keener, 
    courage the more, as our might lessens. “

  3. people who either have no home phone or use cell phones are typically not being sampled. These people are disproportionately young and/or poor (both demographics favor Obama). 
    I remember when all the LewRockwellites were denouncing polls that showed low support for Ron Paul, claiming that internet polls had greater reliability due to the cell phone issue. Ron Paul ended up getting just what the pollsters said he would. 
    Some other yahoo wrote a long post about how The Left has completely distorted our perception of who the likely winner will be in a post Dave Weigel called “a great big wedding cake of stupid“.

  4. Several pollsters actually include cell phones, so I don’t think it has a big effect. I mainly brought it up because it’s comparable to the enthusiasm gap between McCain and Obama in slightly biasing the overall polling by perhaps a couple of points in either direction.

  5. Look, there’s a right way to deal with a loss, and it isn’t retreating into a fantasy world:  
    for retards life is one big fantasy. a postmodern cornucopia if you will.

  6. “Look, there’s a right way to deal with a loss, and it isn’t retreating into a fantasy world” 
    The real world is full of surprises. It’s not rational to assume the worst, even when the “odds” are against you. You should know about that (believe me, I do). 
    (I put “odds” in quotes because we’re not talking about real chance, but uncertainty due to lack of information. There is no way to calculate this kind of “odds”.)

  7. The real world is full of surprises. It’s not rational to assume the worst, even when the “odds” are against you. You should know about that (believe me, I do). 
    that has nothing to do with this thread. the wizbang-retard is in denial of the most likely outcome and is making stupid stuff up (or, is preening his ignorance, as detailed in ben’s post).  
    now, as NH primaries shows us polling is not always right, so it is not the inevitable outcome. that being said, the reasons wizbang-retard gives are pretty mostly false or specious. the polling companies supposedly studied what went wrong in NH, but for whatever reason they haven’t gotten around the releasing the results. it seems possible that they actually don’t know what happened, but don’t want to admit that. sometimes that’s all you can say with any confidence. the dozen polls that are tracking the race right now might be off for a host reasons. if you want the outcome that isn’t being predicted, just hope for it. 
    concocting bizarre conspiracy theories, taking the outlier poll as a sign of “hope,” or highlighting that the *nickelodeon* poll shows the race tied just makes one look foolish. that being said, perhaps that’s the sort of hope people want to have, they need something concrete. even if that concrete thing is a poll of elementary school age kids. 
    there’s no excuse for not distinguishing between the possible and the probable, and the probable and the inevitable. 
    p.s. i don’t get the “it’s not rational to assume for the worst.” i mean, i’m not trained as an engineer, but it seems that it is generally rational to assume for the worst in many situations so you can take the appropriate mitigating measures. or, if you run a small business, you shouldn’t always assume best-case scenarios in terms of revenue so you don’t go overboard on capital inputs.

  8. that has nothing to do with this thread 
    I suppose so, neither does “there’s a right way to deal with a loss”, as the loss hasn’t happened (yet). 
    In any case, as Ben says, the pollsters themselves are worried about systematic bias – which happens all the time in the real world. To my way of thinking about GNXP, that is the relevant topic, not how stupid some blogger might be.

  9. i don’t get the “it’s not rational to assume for the worst.” 
    What I mean is that it’s not rational to assume that the “most likely” outcome is the one that will happen.

  10. What I mean is that it’s not rational to assume that the “most likely” outcome is the one that will happen. 
    depends on the shape of the distribution, right? there’s a mode. and then there’s a mode….

  11. page down to the list of polling data by date: 
    2008 vs. 2004. bush was leading in 2004, but obama is leading a lot more. i think this explains the greater magnitude of right-wing craziness in concocting theories. the left-wing had a greater probability of beating the odds, so ruy teixeira sounds a lot less crazy in his rationalizations because he had a few polls to hold on to ’till the end.

  12. I don’t like the ad hominem tone of Razib’s critique of what was IMHO a very interesting and informative post by DJ Drummond.  
    He (if it is a male) laid his political-interest cards on the table very honestly and provided a lot of relevant data and background.  
    The point is surely that polls near to elections are not the same as elections – they differ systematically in _many_ ways. People poll and vote using different criteria and methods – and it is interesting to explore ways in which polls and elections differ. And polls weeks or months remote from elections are even less like elections than just-before polls.  
    The real craziness is the majority (?) of people including pundits who apparently assume that polls done weeks/ months before the election will predict the election result.  
    Beats me why Drummond should get hit with a diatribe of GNXP invective for doing exactly what good bloggers are supposed to do. Give the guy some respect!  
    I say – Thanks for your post DJ Drummond. I’ll be bearing in mind your arguments over the next couple of weeks.

  13. When I see the word “fisking”, MEGO. It never, ever heralds a valuable use of anyone’s time. Ever.

  14. I think you are exactly right and his arguments are wrong. Nonetheless there is the problem that empirically the polls seem to have overestimated Democrat support most of the time relative to the actual vote count in the last half dozen presidential elections or more. The last election seems to have been the most accurate but even then you had the problem with the early exit polls. Even in the Dole drubbing, Dole’s actual vote count was higher than the poll averages a few days before. So I wonder if it’s a problem in the sampling methodology?

  15. Pollsters wrongly predicted conservative defeat in the UK a while ago (might have been based on an exit poll), the explaination they came up with was that conservative voters are more likely to mislead them. As there is a suspicion attached to saying you won’t vote for Obama there might be something in the idea that the polls overstate his lead. It is not obvious to me this would be good for him. A respected poller claimed the media were understating the lead he had over Hilary. 
    “fun to see him arguing“ 
    Being tested for prostrate (or breast) cancer is not always a straightforward decision for informed people. The testing may well turn up certain relatively innocuous cancers that could be better left alone but oncologists will not do that.

  16. not gandhi, 
    Gore was underpolled by 3 points in 2000, compared to Bush being underpolled by 1 in 2004.  
    Looking through past polls I see you’re right about Dole being underpolled in the last few days. Possible explanations could be last day movement toward Dole, or perhaps issues with Clinton’s soft support not coming to the polls. (I’m too young to have been following that election in-depth, so I don’t know what it could be.) Worth looking into, but I think it’s safe to say that it’s methodological and not a result of pollster political bias.

  17. Probably its not a good metric to measure a poll reliability by the final outcome. Since, the poll can itself have an influence on the final results. Many undecided might get influenced by the poll figures and vote for the leading or trailing candidate based on their psychology. 
    Maybe thats why the pres. campaigns have their own independent polls.

  18. self-reference is fun and all, but why would we want to try and poll an alternate universe where polls didn’t exist?

  19. Beats me why Drummond should get hit with a diatribe of GNXP invective for doing exactly what good bloggers are supposed to do. Give the guy some respect!  
    i’m appalled you found that drivel plausible, and that comment certainly reworks my priors in terms of taking you seriously. polls are imprecise; that does not entail that an elaboration which illustrates ignorance is worthwhile. not at all.

  20. Many undecided might get influenced by the poll figures and vote for the leading or trailing candidate based on their psychology. 
    do you have a cite for this? this is a common perception, and i believe one of karl rove’s theories. but one thing that would to work against this is that undecideds are low information voters, who probably aren’t the types following pollster.com closely ;-)

  21. do you have a cite for this? 
    No, just what seems a plausible speculation on my part. So not going ahead with that to add to the noise :)