The Personal Blog of Stephen Sekula

Beliefs Deserve Better Science

When religious beliefs are placed into the sphere of scientific test, scientific criticism of those beliefs is fair game. All questions put under the framework of the scientific method enjoy the same scrutiny, and it is imperative to approach the question with the same critical toolkit as one would approach any other question. When my issue of The Economist showed up on my Kindle on Friday, I was at first intrigued to see an article suggesting that prayer makes couples more faithful to one another (as opposed, for instance, to just thinking positive thoughts). As I realized that the reporting on this study was done with some carelessness, however, my interest turned to anger at both the incomplete reporting of The Economist and the questionable conclusions of the study’s authors.

The study in question was recently published in The Journal of Personality and Social Psychology [1]. The Economist [2] reported that the study, of 83  undergraduates from Florida State University, showed that praying for your partner reduced the rate of infidelity by a “significant” amount over the 4 weeks of the study. The group of 83 was initially tested to determine their level of infidelity, scored on a scale of 9 points; the average of the entire group was 3.5, with ” . . . [considerable variation] between the four groups.” (as reported by the Economist). A higher score indicates higher infidelity. The students were then broken into four groups randomly, and each was tasked with a different mental exercise. One of the groups was tasked with prayer for their partner. At the end of four weeks, as the Economist reports, “People who had prayed for their partners averaged 2.4, significantly lower than their initial scores, whereas those who thought positively about their partners or considered their day both showed ratings of 3.9—significantly higher” [2].

What does the paper actually say? As a scientist, my first question is: “What does ‘significantly’ mean – that is, what is the variation in each group around these reported means?”

To answer that, I looked at the paper. The results from this study were reported as follows in the paper [1]:

Infidelity. . . . participants in the prayer for partner condition reported significantly lower infidelity scores (M = 2.44, SD = 1.04) than did those in the neutral condition (M = 3.91, SD = 2.16), F(1, 78) = 7.61, p < .01, eta_p^2 = .87, and the positive thoughts about partner condition (M = 3.90, SD = 2.37), F(1,78) = 6.70, p < .01, \eta_p^2 = .80, but not those in the undirected prayer condition (M = 3.19, SD = 2.11), F(1, 78) = 2.02, p < .16, \eta_p^2 .45. However, those in the undirected prayer condition did not significantly differ from the other two conditions ( ps > .05, ns). All the means reported in this and subsequent analyses were adjusted for the covariates (see Table 2).”

The notation of the statistical analysis used here, ANCOVA, is described elsewhere [3]. The F-value and p-value above are as determined from the standard Fisher Test. However, the key here is that the means have uncertainties – standard deviations, or “SD” – which result from the small statistics of the samples (four groups made from 83 students means roughly 21 students per group, which is not a lot of people per group). If I took four random samples of 21 students from a single group of 83 total students, whose total average scores were 3.5 and whose SD for those scores were about 1-2 overall, I could easy generate four samples with this trend. The reality is that there is no statistically significant difference between these four groups.

To review the science here, 83 undergrads were broken into 4 groups. One group had to pray for their romantic partner each day (using a prescribed prayer), 2 had to conduct undirected prayer, and the last simply had to reflect on the day. All groups performed their assigned activity for the same time, once per day. At the end of the study, the mean infidelity score for the directed prayer group was 2.44 +/- 1.04. For the two undirected prayer groups, it was 3.19 +/- 2.11 and 3.90 +/- 2.37. For the neutral group (daily reflection) it was 3.91 +/- 2.16. Nothing can be concluded from this limited data set – these numbers are, within the rules of statistics, all the same. In order to shed any meaningful light on this question, a sample size at least 3-5 times bigger is needed.

The worst criticism I have of this study is that in its present form it CANNOT be conducted blind. The participants were quizzed on their fidelity before being broken into groups. One of those groups was asked to pray for their partner. Those people could easily be consciously or unconsciously influenced to act more faithful over the 4 weeks of the study, since there is likely some sense that their fidelity is being directly tested. There is no way to control for that in this study, and by construction it is potentially biased. At the very least, that bias would need to be quantified.

It’s my opinion that it was irresponsible for The Economist to fail to report the standard deviations in this study and to simply regurgitate the conclusions of the study without thinking about the results. The Economist is a major, international news analysis of serious record. Regarding the study authors, the fact that the study uses a small sample and doesn’t include any attempt at blinding or bias correction suggests this area of research needs more maturity. Until then, nobody should take it seriously.

When people want aspects of religious belief to enter the realm of scientific scrutiny, two things need to happen. First, those conducting the test owe those of religious faith the best possible scientific practice; this study, in my opinion, falls woefully short of that responsibility. Second, those whose faith is, literally, being put to the test should demand better methods but be prepared to expect results at odds with their belief system. After all, nature does as nature does, no matter what you or I believe. At the very least, inadequate studies like this only serve the confirmation bias, rather than real scientific discovery.

[1] “Faith and unfaithfulness: Can praying for your partner reduce infidelity?” http://www.apa.org/pubs/journals/psp/

[2] http://www.economist.com/node/16886238

[3] ANCOVA: http://udel.edu/~mcdonald/statancova.html

2 thoughts on “Beliefs Deserve Better Science”

Comments are closed.