Blinding. It’s a word that is used in funny ways in science. “We performed a blind analysis.” “We blinded the data until we had finished our background studies.” “We fit for the background, extrapolating into the blind region.” “Blind” is used as a verb, adjective, noun, and just about every other part of speech to describe a particular approach to conducting an analysis. Today, the MiniBooNE experiment revealed to the world the results of its long awaited, now unblinded, analysis.
The best example of a “blind analysis” that I can give is a more down-to-earth situation: drug tests. Let’s say you want to test the efficacy of a “natural remedy.” One way to do this would be to give 10 patients the remedy and 10 patients a placebo. As the doctor or scientist, you will know which patient gets which pill. However, studies of this approach have shown that if a researcher or doctor wants the pill to work badly enough, they might accidentally given subtle clues to patients who get the real drug, or give the real drug to people whom they think might be more sensitive to its effects, thus introducing “bias” into the measurement. As a result, you might find positive results for a drug which in reality is no better than a placebo.
A blind approach is, instead, to give the real drug pills to a third party, who prepares samples for each patient. The third party labels each bottle of pills with a code, and keeps a private record of which labels correspond to the real drug and the placebo. The bottles are given back to the doctor, but the relationships between bottles and pills is kept secret. Thus, the patients won’t know what they’re getting and the doctor won’t know who is getting the real medicine. Interactions between patients and the doctor are unbiased. After the regimen is complete, the doctor tells the third party which patients got which labeled bottles, and they look for a correlation of benefit with the real drug. That’s considered a blind, unbiased analysis.
In physics, you can do the same. You can define a region of the data where signal would probably like to live, and never look at it. To test how good your understanding is of all the non-signal stuff in the data, you define “control” regions around the signal region that let you test your assumptions. Once you think you understand those control regions, you can then “unblind” and see if your results match the signal, or non-signal, expectation.
This is what MiniBooNE did. In the process of being blind, they learned a lot about neutrino events, and non-neutrino backgrounds that pollute their results. They satisfied themselves over a period of years that they understood the control data. They had two independent analyses of the same data, to avoid a single, potentially biased method of analyzing the data (“groupthink”). In the end, they appear to rule out the old LSND sterile neutrino result, with only a 2% chance left that LSND is right.
Let me make a statistical aside. 2% is acually a big number in science. If you repeated MiniBooNE 1000 times, and analyzed the 1000 MiniBooNE data sets with the same analysis, you should find the results scatter according to statistics as a Gaussian distribution, with a “width” that encompasses 63% of the experiments. That’s called “1 sigma”. Two sigma would correspond to 86% of the experiments, 3 sigma 95%, and four sigma 98%. Therefore, to be only 2% compatible means that you’re at about 4 sigma from the LSND result. As a friend of mine likes to say, “Five sigma happens all the time!” Why? 2% of 1000 experiments is 20 experiments. That means that MiniBooNE could have just been unlucky, and thinks they’ve ruled out LSND when, in fact, with more data they’d find out they didn’t.
Of course, that’s unlikely.
Confusing? Well, that’s the world of probability. The real mystery is the other result that came out of MiniBooNE. At very low neutrino energies, they find an excess of events that cannot be explained by their non-signal, nor their current signal, hypotheses. What could this be? Some unforeseen background? Some new, unexpected signal? Only more work will tell.
Powered by ScribeFire.
