A lot of people get a little time on the radio, or on the internet. In that time, they often pitch a premise, and based on that sell an idea. Much of what is out there is crap. This blog might count. Regardles, when you’re presented with a radical idea that is based on a “scientific” premise, remember: study the premise.
Science is a critical tool, available to all people, that allows us to comprehend the natural world. Since people are part of nature, it is therefore a critical tool to understand human behavior. The theory of evolution has already taught us a great deal about social structure, about the role of selflessness and selfishness to the functioning of society. It has given us clues about the origin of language, and controversial ideas about why humans are largely hairless creatures.
A survey is an important aspect of the science toolkit. Properly constructed, with control questions that allow the interviewer to judge the sincerity of the interviewee, a survey is a powerful tool that can teach you about your fellow humans. However, a badly constructed survey (very easy to do) can lead to misleading conclusions, and a badly interpreted survey is even worse. In the hands of the unqualified, those who haven’t studied the art of constructing a survey, this is a dangerous tool that rings of science to the untrained listener, but whose sound is murky and hollow.
This past week, I was treated to the results of a badly constructed survey, badly interpreted by a person with an agenda. “All Things Considered” ran a story on the “Conservapedia”:http://www.conservapedia.com/Main_Page. Conservapedia is a “conservative” answer to the Wikipedia:”http://www.wikipedia.org”. The link to the story is here:
“http://www.npr.org/templates/story/story.php?storyId=8286084”.
The founder of Conservapedia, Andy Schlafly, laid out the argument for such a juxtaposition. Wikipedia editors, he says, are more liberal than mainstream Americans. Therefore, Americans aren’t getting a web encyclopedia reflective of their views and values. Conservapedia answers this problem. We see his premise: data tells us that wikipedia editors are not in line with mainstream America. Therefore, mainstream American needs to be served and we need Conservapedia.
What is the basis of this premise? In the story, he points to an unsourced survey that says that Americans count themselves as “conservative” 2-to-1 over “liberal”. He never defines those terms. In contrast, he says, wikipedia editors count themselves 3-to-1 liberal over conservative.
A good citizen, armed with science, sees flaws in this premise. First, what survey is he quoting? How were the terms defined, and how were the questions phrased? Were control questions used? Was the same survey asked of general Americans as of Wikipedia editors? How was the sample determined? Were Americans surveyed state-to-state proportional to their relative population contribution to the U.S., or were they uniformly sampled from across the U.S.? How many of the people interviewed use Wikipedia, or at least use the internet to get information?
In contrast, an easy-to-find study conducted by the Pew Research Center used a series of issues that tend to distinguish “liberals” from “conservatives” (“http://pewforum.org/docs/index.php?DocID=150#1”:http://pewforum.org/docs/index.php?DocID=150#1).
They find (after an exhaustive analysis of the questions, answers, and self-perception) that the breakdown is varied. 12% of respondents are firmly conservative (taking that stand on all five issues). 16% are mostly conservative (4/5 questions), 34% are right in the middle (3/5), 16% are mostly liberal (4/5), and 22% are firmly liberal (5/5). Taking “mostly” and “firmly” together, we find that of those who answered in one framework 4 or more questions, there are 11% more liberals than conservatives. The rest of people are right in the middle (3/5), and can’t really be counted as one way or the other.
So where did Mr. Schlafy get his data? He didn’t cite a source, so we’ll never know. But a comprehensive and detailed survey, whose assumptions and questions are laid out for all to see, conclude that of those who are mostly liberal or conservative it’s about 1:1, with the rest being somewhere in the middle. And, since we don’t know what the basis of his Wikipedia numbers are, we can’t compare. The premise is flawed, and the conclusion should be ignored. This is a great example of bad science.
But there is another flaw in Mr. Schlafy’s argument. This one falls under “bad interpretation”, and has to do with the idea of “normalization”. When two fractions have the same denominator, we say that they are “correctly normalized” – that is, they come from the same underlying sample (denominator) and thus can be compared directly to one another. For instance, if you ask 100 people the same question, and 10% answer yes, 20% answer no, and the rest answer “maybe”, you can definitely say that people say no 2:1 to saying yes. However, what if you ask two different groups of 100 people the same question, and in one group 10% say yes, 30% say no, and the rest say “maybe”; in the other, 45% say yes, 20% say no, and the rest say maybe? A bad interpretion of this data i that people say no 2:1 to saying yes.
Why? The normalization is different. You can’t compare two different groups without understanding how similar or different they really are. Therefore, you can’t say people respond no twice as much as yes if you are comparing 20 people in one group to 10 people in a different group. That’s also bad science.
This is also precisely the mistake made by Schlafy. Why? Well, the set of people who are Americans, and the set of people who are Wikipedia editors, are not the same. Wikipedia is open source, meaning anybody can be an editor. That means editors come from South Africa, China, Norway, Brazil, Indonesia . . . everywhere. Americans come from, well, America. Therefore, the normalization of his number – 2:1 and 1:3 – are different, and they cannot be compared. Wikipedia serves all of humanity, not just Americans. Of course it’s not necessarily in-line with American views. It’s probably not in line with Chinese views, or South African views. But that’s not what matters. What matters is that anybody can correct it if it’s wrong, and cite their source.
This last point was made by a very astute listener to “All Things Considered”, who wrote a letter to the program: “http://www.npr.org/templates/story/story.php?storyId=8939263”:http://www.npr.org/templates/story/story.php?storyId=8939263.
