Mike Berman’s Washington Watch

February 17, 2012 11:58 AM

There Are Polls and Then There Are Polls

When I began absorbing political surveys the highest state-of-the-art method had polltakers going door-to-door, having direct one-on-one conversations with respondents.

Over time that changed, and the highest state-of-the-art method became and still is the telephone survey, with a direct one-on-one conversation between a questioner sitting in a phone bank and a respondent. Until recent years, those telephone conversations were on landlines. Now, because of the number of people who only use cell phones, a proper sample includes cell phones as well as landlines.

Various techniques have been designed for creating random samples that are likely to approximate a particular group. That group might be all adults, only those who are currently registered, or only those who say they are likely to vote in the election that is the focus of the survey.

Over the last few years, a number of polls and polling organizations have come into play that use a variety of “new” techniques whose “efficacy” remains to be seen. These include wholly automatic telephone surveys and on-line surveys.

The moral of the story is that, when considering survey research, published or private, you need to take into account the methodology used in conducting the research. This is especially true when the result is out of whack with other surveys of the same group during roughly the same time period.

The following are items to which you might pay attention when evaluating a survey:

* Is it conducted online and, if so, how does it purport to create a random sample? Random samples can be developed online when the universe is fixed, for example when surveying the membership of an organization for which the email addresses of the members are available. Otherwise, no good device yet seems obvious.

* Is it conducted among registered voters or likely voters, and how close is it to the election? Likely voters makes more sense closer to election day.

* What is the size and nature of the sample? Is it smaller than the sample of other polls surveying the same landscape? Small samples often result in less than reliable results.

* Are all potential participants included in the sample? Nate Silver cited an example of a survey taken for a primary election that did not include Independents when they are free to participate in the partisan primary.

* If it is a telephone survey, are an appropriate number of cell phone respondents included?

* Is it clear as to the time period in which the survey was conducted, as opposed to the time at which it is published?

* Is information about the survey methodology, sample size, and timing of the survey prominently displayed with the results?

There are a number of aggregators of national and State surveys. They provide an invaluable service, making it unnecessary for consumers of this kind of data to scour various sources for the latest information.

However, to the extent that such services calculate an average number for some time period, there is a need to pay attention to which surveys are included in the average.

An average that includes with equal weight a survey taken by NBC/WSJ and an automated telephone survey that does not include cell phones needs to be viewed and used with care.

The following are the published surveys and survey research organizations with which I am comfortable without further analysis. This is not to say that other entities don’t do credible survey research.

PEW Research
PPP – Public Policy Polling
Survey USA
Garin-Hart-Yang Research Group
Greenberg, Quinlan, Rosner
Winston Group

When considering comparisons of survey results from different time periods it is important to remain mindful of the context.

As the election approaches, one of the popular comparisons is the relevance, of a sitting President’s job approval to the prospects of re-election. The test seems to be whether the incumbent President is at or above the 50% mark. Of course, George W. Bush was reelected with a 48% approval rating. And, there have been only 11 Presidents who sought re-election since the advent of polling. It is not much of a sample, but it is all there is.

In a recent Washington Post article about the Democrats’ prospects for retaking the House, the writer makes the point that the Republicans have had a margin of this size 5 times since 1900. Each time they had huge losses in the next election. To the credit of the writer the article goes on to make the point that the Republicans have not had a margin of this size since the 1940. A lot has changed since then. [WP 2/7/12]

Return to Home Page