Mention the name Literary Digest to a pollster and they will instantly know what you are talking about. Literary Digest is well-known for their famously wrong prediction that Kansas Republican Alfred Landon would beat Franklin Delano Roosevelt in the presidential election of 1936. Part of the problem was that, despite a sample size of 2.4 million and a response rate of nearly 25 percent, the groups that Literary Digest surveyed were not representative of voters. Respondents tended to be wealthier than average, since they were drawn from the Digest‘s subscribers as well as automobile registries and telephone books. Using a sample of “only” 50,000, George Gallup was able to predict the outcome correctly and the Digest soon went out of business.
What people forget is that 1936 was not the first time that Literary Digest had conducted a presidential poll or made a prediction. In the previous four elections–dating back to 1920–the Digest had always been correct. The 1936 election was a “falling off the cliff” moment for their polling methodology.
On Friday David Rothschild of Microsoft Research came and gave a series of talks for the Duke political methodology group. He covered a number of interesting topics, including prediction markets and online experiments. There was also a presentation about his work-in-progress analyzing 2012 polling data collected via XBox Live. One takeaway from that presentation is that, correcting for demographics of likely voters (as you might expect, XBox respondents were overwhelmingly male and young) the Xbox Poll tends to track the Pollster polling average.
An important issue that came up during the presentation was non-response bias. Telephone surveys now have vanishingly small response rates. They are further complicated by the shift to cell phones. Pollsters cannot use a computer to randomly dial (RDD) cell phones: the numbers have to be dialed by hand, which raises the time required and thus the costs of the poll. People are not “randomly” switching to cell phones either, so this biases the poll.
The demise of telephone polls will not be gradual. Organizations like Gallup will have their own Literary Digest moment in which their methodology–which has been highly accurate for years–will fall off a cliff. It is only a matter of time.
Matt Dickenson 