The Science of "Disestimation"; December 2010; Scientific American Magazine; by Charles Seife; 1 Page(s)
At the end of September the Pew Forum on Religion and Public Life released a survey that seemed to show that nonbelievers knew more about religion than the faithful. Some media outlets crowed about the results (“Atheists Know More about Religion Than Believers,” Time magazine declared), whereas others turned to comforting the faithful (“We Didn’t Flunk the Religion Test,” FoxNews.com insisted). Few seemed to realize that the polls were far from immaculate. In fact, the episode was a good example of what I call disestimation: the act of taking fuzzy numbers way too seriously.
At first, it might seem like a cut-and-dried story: out of 32 quiz questions, atheists and agnostics, on average, got 20.9 correct, higher than any other group and higher than the overall average of 16.0 questions right. But because Pew managed to reach very few atheists and agnostics—only 212 people out of the 3,412 included in the survey—the 20.9 number masks a tremendous amount of imprecision. Small samples don’t give reliable numbers, and if you present the poll results using a standard graphical technique to represent uncertainty (below), you can see that the distinction between atheists/agnostics and Jews and Mormons evaporates.
