Explaining How Polls Can Mislead

I've often said that polls are only good for identifying a general trend rather than specific information, and a couple days ago I mentioned hearing about how polls can be slanted to show just about anything.  Well, guess what?  I finally tracked down some information that explains a couple ways that can work.  This was an education for me, so I thought it would be worth passing along to you.

Here's the first example, from ABC.  Read this headline like normal, as if you were scanning a newspaper:

Voters' Economic Jitters Shake the Race in Virginia

Obama Holds 10-Point Lead Over McCain in Trust to Handle Economy

Now, without looking back at the headline, how far ahead is Obama when it comes to the race in Virginia?  10 points, right?  Wrong.  If you look back, you'll see that the 10-point lead is only on the economy question, not the whole race.  It is at the bottom of the first page of the article that they finally report the overall race is only three points apart.

Here's how it works.  ABC sets the topic of the article with the headline, telling the reader they're talking about the overall picture in Virginia.  The reader immediately sees that Obama has a 10-point lead, therefore assuming that lead is in the overall picture.  But, at the very end of the sub-heading, ABC changes the topic of the story to be about the economy, not the overall race.  Even if the reader catches the abrupt change in topic, the article still buries the true state of the overall race at the bottom of the page.

They're presenting the information accurately, but in a misleading way.  It's hiding in plain sight.  And don't give me any pushback on reading the whole thing carefully...you know how often people just scan headlines to get the basic idea of the article, so this is a completely valid criticism.

The second example is much more subtle, and pretty much impossible to catch without some deep digging.  Rush Limbaugh had mentioned this on his radio program recently, so I went to the website he referenced to get all the details.  First, we have a recent Gallup daily tracking poll with this headline:

Gallup Daily: Obama 48%, McCain 44%

That's pretty typical of how the race has been for a while now, right?  Wizbangblog does the analysis of the real story.  Stick with it, because it's really interesting stuff that can explain a whole lot about polling!

Let's start with the latest poll numbers. Yep, Obama back on top, is the headline for many of them, though it's a bit tight. I guess we should worry on the Right? Hmmmm, well maybe not so much, just as those on the Left did not have that much to worry about when McCain got the 'Palin Bounce' earlier this month. I said when the first bumps came out that I did not think Palin's effect would really be that immediate, and I have always said that the reader should go well past the headline to find out what a poll says. So, taking my own advice, let's see what Gallup has to say.

The Gallup Organization is as clean and straight-arrow a polling group as I have ever found. Their methodology is consistent and transparent, their questions are the same and they have a longer history than anyone else in the business. But even Gallup has a few odd quirks, and when you see them it might change how you look at their poll releases. For this article, I am looking at the Gallup 'Daily Tracking Poll' for the Presidential election. For the five most recent weekly reports, here's where Gallup says the candidates stood:

Aug 21: Obama 45, McCain 44
Aug 28: Obama 48, McCain 41
Sep 04: Obama 49, McCain 42
Sep 11: McCain 48, Obama 44
Sep 18: Obama 48, McCain 44

From that, it appears that a tight race opened up first for Obama, then McCain, then Obama again, with each candidate sitting anywhere from 41 to 49 percent support (not counting margin of error) during that time.

So how do we explain that strange and short-lived reversal?  It would help to track it on a smaller, more detailed scale, like this:

[L]et's look at their support by party identification, first by Obama:

Liberal Democrat support for Obama - 88% Aug 21, 91% Aug 28, 93% Sep 4 through Sep 18.

Moderate Democrat support for Obama - 78% Aug 21 and 28, 81% Sep 4 through Sep 18.

Conservative Democrat support for Obama - 68% Aug 21, 63% Aug 28, 77% Sep 4, 70% Sep 11, 66% Sep 18

Hmmm. Obama's support goes up and down, but the Liberal and Moderate Democrat support for Obama has been steady all of September. Odd, isn't it? And support for Obama among Conservative Democrats went down four points in the last week, even though his overall support is supposed to have gone up four points. How to figure that?

Perhaps it's in the Independents. After all, if Obama started winning them over, he'd not only be making gains overall but gaining support where he wants it the most.

Independent support for Obama - 24% Aug 21, 29% Aug 28, 23% Sep 4, 29% Sep 11, and 27% Sep 18

Hmmm, again. Obama gained support among Independents in the last month, but he actually lost two points among Independents in the last week. So that 4 point gain overall is still a mystery.

Nothing to do, then, but look at the Republicans. It would really be something if he's improving support from GOP voters:

Liberal/Moderate Republican support for Obama - 16% Aug 21, 13% Aug 28, 14% Sep 4, 16% Sep 11, 10% Sep 18

Ouch. Obama lost six points among Liberal and Moderate Republicans in the past week.

Conservative Republican support for Obama - 6% Aug 21, 5% Aug 28, 4% Sep 4, 3% Sep 11 and 18

No change there in the past week.

Taken altogether, there is no group of political identification where Obama's support has increased in the past week.

Before going into the explanation Wizbang runs a similar analysis on McCain's numbers:

The man, according to Gallup, lost four points of overall support in the past week,

Conservative Republican support for McCain - 89% Aug 21, 91% Aug 28, 94% Sep 4 and 11, 95% Sept 18

Interesting. McCain's support among Conservative Republicans went up a point in the last week. Well, let's move on:

Liberal/Moderate Republican support for McCain - 75% Aug 21, 77% Aug 28, 78% Sep 4 and 11, 85% Sep 18

Wow, McCain's support from Liberal and Moderate Republicans climbed by seven points in the past week, and yet we are told his overall support fell by four points? That is very odd, wouldn't you say? It must have been the Independents, perhaps?

Independent support for McCain - 34% Aug 21, 31% Aug 28, 29% Sep 4, 28% Sep 11, and 32% Sep 18

Stranger and stranger, McCain's support among Independents went up by four points in the past week, just as his support from Republicans increased, yet we are told his overall support went down by four. Very hard to explain that using the math most of us learned in school, isn't it? Well, there's just one place left to look. Maybe somehow McCain used to have significant support among Democrats, but lost it? Let's find out:

Conservative Democrat support for McCain - 23% Aug 21, 26% Aug 28, 15% Sep 4, 21% Sep 11, 24% Sep 18

Hmpf. Once again, a group where support for McCain went up, but the overall says he went down.

Moderate Democrat support for McCain - 14% Aug 21, 13% August 28, 11% Sep 4, 12% Sep 11 and 18.

Steady there, so that one does not explain it.

Liberal Democrat support for McCain - 6% Aug 21, 6% Aug 28, 4% Sep 4 and 11, 5% Sep 18.

It's only a point, but again we see McCain's numbers in this group went up.

So, put it all together, and in the past week Obama has stayed steady or lost support in every party identification group, yet Gallup says his overall support went up four points. And McCain stayed steady or went up in every party identification group, yet we are supposed to accept the claim that his overall support went down by four points? Anyone have an answer for how that is even possible?

If you're like me, you're scratching your head.  Scratch no more, because here's the key (bold emphasis mine):

There is one, and only one, possible way that such a thing can happen mathematically. And that way, is that Gallup made major changes to the political affiliation weighting from the last week to now. Gallup has significantly increased the proportional weight of Democrat response and reduced the weight of Republican response. Bear in mind that this assumes that people change the foundation of their political opinion like a showgirl changes costumes, which has no scientific basis or historical support whatsoever. As I said earlier, the Gallup Organization is very much a professional polling agency, who tries their level best to gauge the national mood. That, after all, is why I chose to use their poll for my examination. I could do the same thing with any other of the major published polls, and I can tell you straight-up that I would find the same practice going on everywhere. But just because something is popular, does not validate it as a scientific method. Rather than report the rising and falling levels of support for Obama and McCain with constant party identification weighting, the Gallup and other polls are shifting the party weights over time, which pretty explains how the 'bounce' happens for each convention. ... The problem there, is that such weighting is still very subjective, and what's more it fails to consider that someone may consider themselves a member of one party with respect to the House and Senate races, but something else entirely when it comes to voting for the President. The state of Oklahoma, for example, is a very Democratic place, but it's pretty solid for McCain, just as it was for Bush. So weighting a presidential poll for party identification on the basis of how they think someone will vote for Congress, is going to miss the mark.

So there it is.  That's how you see these wildly shifting poll numbers in the span of a very short time.  Now, what are we to make of this new information?  The main thing, I think, is to view any single poll that you see or hear about with a load of skepticism.  It is only when you see a general shift in repeated polls from multiple organizations that you can begin to see a true shift in 'the polls'.  RealClearPolitics has the right idea with their system of measuring - they take an average of all the other major polls.  It can still be flawed by those major polls' own weighting, but any one poll's weighting flaw will be minimized, and the RCP average is a guage of the general mood rather than a single specific poll.

But, Wizbang doesn't stop there.  In a rather ingenious move, they go back and apply the real results from past elections in an attempt to remove that weighting shift in the current poll:

[I]f we go back and look at the historical track record for the last ten years in terms of self-identified party affiliation from actual exit polls, we see a clear standard of weights; 38.4% Democrat, 35.8% Republican, 26.0% Independent. If we then work them out to fill the liberal/moderate/conservative slots used by Gallup, the following weights have historical validity and may be used as a constant for poll responses:

Liberal Democrat 9%
Moderate Democrat 16%
Conservative Democrat 13%
Independent 26%
Liberal/Moderate Republican 23%
Conservative Republican 13%

If we apply those weights to the poll response, here is what happens to the Gallup polling responses:

August 21: Obama 39.94%, McCain 43.43%, Undecided 16.63%
August 28: Obama 40.04%, McCain 43.60%, Undecided 16.36%
September 4: Obama 41.06%, McCain 41.77%, Undecided 17.17%
September 11: Obama 42.04%, McCain 42.45%, Undecided 15.51%
September 18: Obama 39.62%, McCain 45.71%, Undecided 14.67%

Did you catch that?  According to these historically-based weightings, McCain is actually ahead by over 6 points, a number which is well above the margin of error.  Some other conclusions:

Movement still happens in both sides' support, but it is more gradual and is consistent with events in both parties. Frankly, it is only reasonable to expect that Democrats largely support Obama while Republicans largely support McCain, and even now there is a significant amount of indecision; between one of six and one of seven voters are not sure who they want. Most of that doubt is with independents, whose support may make all the difference in the key states.

The sum effect of all this, is that both Barack Obama and John McCain are gaining support, shoring up their party support and looking for persuadable independents. But McCain is gaining support faster than is Obama, and one might wonder why.

Wizbang goes on to point out that liberal 'trolls' -- jerks, basically -- who have been sent out by Obama to wreck the McCain campaign have actually done the opposite, undermining the ability of Obama to solidify his base and turn to attracting the all-important independents (case in point: all the vicious and outlandish attacks on Palin, McCain's war injury, etc.).  McCain, on the other hand, is succeeding very well.  His pick of Sarah Palin almost immediately brought his base back on board, allowing him to focus on the independents, which he appears to be picking up steadily.

Remember how I said yesterday that I don't think the race is nearly the 'dead heat' everyone thinks it is?  This is why.

In addition to a great analysis of why Obama (the challenger) is not leading by double digits in this time of an unpopular incumbent President associated with his opponent, a long and unpopular war, and a struggling economy, Wizbang also reveals how polls are inherently inaccurate despite the best efforts of polling agencies to achieve accuracy.  I'm sure that Gallup, Zogby, Rasmussen, and other major polls do not intentionally slant their weighting to help (or hinder) any particular candidate, because their reputation and entire business existence depends on accuracy.  On the other hand, I don't believe for a moment that agencies like ABC, NBC, CNN, or the New York Times would hesitate to do so.  Their business is generating sales, and they (along with most of the other media outlets) have openly professed their adoration for Barack the Obamessiah.  Shedding their pretense at objectivity, it is in their best interest to do whatever they can to help him win, even if it means a couple extra points of poll weighting that will be almost totally invisible.

So, keep all this in mind as you hear about individual polls showing something unusual, or when you see polls being used to generate headlines.  Skepticism is the key, and don't get too wrapped up in any one poll.  If you see a whole lot of polls moving in the same direction, THEN you can believe it.

Hope that helps!  It sure opened my eyes.

There's my two cents.