This week offered America a rare but important lesson in statistics. What worried me was that the venue most people get the information from – the news media – widely seems to flub the lesson. I’d give agencies like NBC and CNN a D- in their ability to report accurately on statistical results.
The lesson was thanks to a comprehensive study on the direct physical medical benefits of prayer [LATimesPrayer]. A sample of 1800 patients were evenly divided into three groups. According to the LA Times story mentioned here, and other news agencies reporting the story, the groups were ” . . . those who were told someone was praying for them; those who were told only that someone might pray for them and got prayers; and those who were told someone might pray for them but received no prayers.” According to the article, 65% of those in the study professed a belief in the power of prayer. Each patient then underwent heart surgery. Monastaries of several faiths were asked to conduct the prayers, and the study was done “double-blind”, meaning that nobody involved knew enough information to bias the outcome, and the correlations were only known once the data were assembled in the end.
And what were the results? Well, after observing the patients’ recovered for the next 30 days, “59% of patients who knew they were being prayed for had complications, compared to 51% of the patients who did not receive prayers”.
Several news agencies (not the LA Times) then reported that it seemed that being prayed for introduced complications! Witness 8% more people with complications in the group prayed for. But is that really an accurate conclusion?
The answer is an emphatic NO. Being a scientist means appreciating the limits of your data sample. Here, we have 1800 people divided into three groups. That’s 600 people per group. Now, since each individual is uncorrelated with other individuals (e.g. their individual surgeries do not affect each other, nor do they affect each others’ recoveries, nor do they have things in common like the surgical team), we can treat the uncertainty on their population with “Poisson statistics”:http://en.wikipedia.org/wiki/Poisson_distribution. Poisson statistics is a simple tool by which you can take a large sample (greater than 10-20 people, in this case) and determine the affect that random variations will cause in “smearing out” the conclusions you can draw from studying the population.
The Poisson uncertainty – the limit at which we can interpret numbers before we reach the inherent random fuzziness in the data – is given by the square-root of the population. Therefore, for the WHOLE group of 600 people, the limit of our knowledge is 24.5 (the square-root of 600). That is, if results between groups of 600 people differ by less than 24.5 individuals, they are not statistically significant and you cannot say whether the results indicate differences in the population. 24.5 people corresponds to 4.1% of 600 people.
Now, knowing this let’s study the results of the prayer experiment. In one sample 59% of people suffered complications – that corresponds to 59% of 600 people, or 295 people. The uncertainty on this result is given by Poisson statistics: the square-root of 295 is 17.2 people, or 5.8%. Therefore, all we can say is that somewhere between 53% and 65% (rounding the results) of people in this sample should have expected to suffer complications as a result of the heart surgery.
What about the sample that wasn’t prayed for? Is 51% compatible with 59%, indicating there is *no difference* between being prayed for and not being prayed for? Well, let’s run the exercise and see if these numbers (59% and 51%) agree within the Poisson uncertainty – if they do, then all we can conclude is that prayer isn’t a factor, and both groups will suffer the same rate of complications. 51% of 600 is 255 people, with a Poisson uncertainty of 16 people, or 6.2%. That means that of the people in this group, between 45% and 57% should have expected to suffer complications. As you can see, (51 +- 6)% overlaps well within the Poisson error of (59 +- 6)%. Therefore, all we can conclude is that prayer is not a mitigating factor in reducing complications. It neither helps, nor hurts.
What is striking to me is that the press missed the one overwhelming significant result of this study: if you have heart surgery, you have a 50% chance of suffering complications.
Prayer or not, heart surgery is rough business.
.. [LATimesPrayer] “http://www.latimes.com/news/nationworld/nation/la-sci-prayer31mar31,1,3169049.story?coll=la-headlines-nation”:http://www.latimes.com/news/nationworld/nation/la-sci-prayer31mar31,1,3169049.story?coll=la-headlines-nation
