The election to fill the late Ted Kennedy’s Massachusetts Senate seat captivated the nation not only because much in the Obama Administration’s agenda was at stake, but because the polls seemed inconclusive and inconsistent. A week before the election, some polls were still showing Democrat Martha Coakley well ahead, but a surge in support for Republican Scott Brown was becoming evident. Polling organizations make strong scientific claims for their results, but more than 70 years after George Gallup revolutionized polling it is still as much an art as a science.
“Uncanny accuracy.” “Conclusive evidence.” “An achievement of no little magnitude.” Back in the 1930s, The Literary Digest was coyly (and disingenuously) disavowing such claims for its presidential polling. But the respected news magazine had correctly predicted election outcomes from 1916 through 1932, and few expected anything different in the 1936 race between President Roosevelt and Republican challenger Alf Landon.
The Digest’s final poll results, printed on Halloween, predicted a big loss for FDR.
Every week hundreds of thousands of readers flipped through the pages of the venerable Digest for reporting and analysis of current events. Among articles on politics, foreign policy, and culture, Americans also saw ads for iconic brands like Remington typewriters and GM cars, along with more ephemeral products like Duofold Health Underwear, Marvin’s “Ambassadors” Cigars (50 for $1.70), and the self-help philosophies of the day (“Psychiana—an absolutely new understanding of Life and God, as modern in its application as radio or air travel”).
Print ads in 1936 looked much as they do today. But behind the scenes, a journalism professor, ad man, and columnist named George Gallup, along with his new American Institute of Public Opinion, was revolutionizing the business of market research—discovering, to publishers’ and advertisers’ surprise, that such factors as typography and page position significantly affected reader response to ads, and that pictures and comics got more of readers’ attention than articles. Now Gallup was applying his survey methods to presidential polling, and what he found made him certain the Literary Digest was dead wrong.
With the country still deep in the Depression, the magazine’s fortunes, like many businesses,’ were declining. Still, its election polls had become something of an institution, and in 1936 it confidently distributed some 10 million cards to addresses gleaned from, among other sources, telephone directories and automobile registrations. Some 2.4 million Americans from all 48 states returned ballots—not a large response rate for a straw poll, but still a huge sample. The odds an adult 21 or older was polled by The Literary Digest about how he or she would vote in the 1936 presidential election are 1 in 33.97. And given the Digest‘s track record, accurate results seemed a foregone conclusion.
But things could hardly have gone more wrong for the Digest and its editor-in-chief, Wilfred Funk. Son of the founder of Funk & Wagnalls, the lexicographer and language maven helmed a respected journal that dated back to 1890. But this time around, he wasn’t even close.
Even today it may seem counterintuitive that a very small sample, however carefully selected, should result in better predictions than a huge straw poll; back in 1936 it seemed inconceivable that Gallup’s survey of 30,000 voters could out-predict a poll about 80 times larger. But that’s exactly what happened. Instead of losing by 14 points (55% to 41%) as the Digest poll had predicted, Roosevelt won, and by the far larger margin of 61% to 37% (Gallup’s poll had actually put Roosevelt even further ahead, with 66%).
Gov. Landon netted only 8 electoral votes. He didn’t even carry his home state of Kansas.
How did the Digest screw up so badly? Modern scholarship points to two main causes. First, the sample itself seems to have been biased. Over-reliance on automobile and telephone records excluded many of the poor, who tended to support Roosevelt but couldn’t afford cars and phones. Second, a greater proportion of Landon supporters responded to the poll. It seemed that voters who were angry at Roosevelt were more motivated to stand up and be counted when the Digest came calling.
Applying lessons learned from their commercial market research, Gallup’s group improved accuracy by conducting in-person interviews, carefully wording their questions, and using “quota sampling” to obtain survey answers from specific subsets of the population, including lower-income respondents. In this way, samples could be kept quite small and still produce results with acceptably narrow margins of error. The odds an eligible voter completed a poll or was interviewed by the Gallup Poll about how he or she would vote in the 1936 presidential election are 1 in 2,495.
Gallup not only called the election right, he added insult to injury by correctly predicting that the Digest would get it wrong, and by how much. The embarrassed editors’ credibility suffered immediately, subscribers abandoned ship, and within a year the hoary journal was gone.
The movement to develop systematic, “scientific” polling methods had been building since the 1920s, rooted in democratic ideals as well as dollars. Gallup was ” convinced that American democracy was seriously threatened by undemocratic leaders who purposely misrepresented American public opinion to promote their own selfish and ruthless agendas.” Against a backdrop of rising dictatorships in Europe and the bitterly controversial New Deal at home, debate about democracy—its nature and its defense—resonated strongly with the public. Gallup believed systematic polling could counter the influence of special interests and curb the power of a potentially dictatorial ruling elite by accurately measuring and reporting the public will.
In a country where democracy and capitalism are inextricably linked, perhaps it’s no surprise that George Gallup’s career and values spanned both commerce and the commonwealth. Ever the populist, Gallup went on to tap into the public’s “collective wisdom” in Hollywood, where his surveys influenced how the studios promoted their films and stars. The Gallup Organization and its competitors grew and thrived. Despite setbacks like the famous “Dewey Defeats Truman” error of 1948, the polling industry became an essential weave in the fabric of society. News organizations and the public voraciously consumed poll results during 2008’s presidential election, did the same in the recent Massachusetts contest, and will certainly do so in the upcoming midterm Congressional elections. Politicians, businesses, and organizations of every stripe depend to this day on the systematic methods pioneered by George Gallup all those decades ago.
The 2010 Census is now getting underway. With Congressional representation and federal funding to be determined, there’s more at stake in this headcount than bragging rights for the best polling method. And the debate around who is doing the counting brings to mind the core questions about public representation raised by George Gallup. The debate has already turned bitterly partisan, as evidenced by Republican Senator Judd Gregg’s withdrawal from the nomination process for Secretary of Commerce in President Obama’s cabinet, citing one of his reasons as the “politicization of the 2010 census.”