When religious beliefs are placed into the sphere of scientific test, scientific criticism of those beliefs is fair game. All questions put under the framework of the scientific method enjoy the same scrutiny, and it is imperative to approach the question with the same critical toolkit as one would approach any other question. When my issue of The Economist showed up on my Kindle on Friday, I was at first intrigued to see an article suggesting that prayer makes couples more faithful to one another (as opposed, for instance, to just thinking positive thoughts). As I realized that the reporting on this study was done with some carelessness, however, my interest turned to anger at both the incomplete reporting of The Economist and the questionable conclusions of the study’s authors.
The study in question was recently published in The Journal of Personality and Social Psychology [1]. The Economist [2] reported that the study, of 83 undergraduates from Florida State University, showed that praying for your partner reduced the rate of infidelity by a “significant” amount over the 4 weeks of the study. The group of 83 was initially tested to determine their level of infidelity, scored on a scale of 9 points; the average of the entire group was 3.5, with ” . . . [considerable variation] between the four groups.” (as reported by the Economist). A higher score indicates higher infidelity. The students were then broken into four groups randomly, and each was tasked with a different mental exercise. One of the groups was tasked with prayer for their partner. At the end of four weeks, as the Economist reports, “People who had prayed for their partners averaged 2.4, significantly lower than their initial scores, whereas those who thought positively about their partners or considered their day both showed ratings of 3.9—significantly higher” [2].
What does the paper actually say? As a scientist, my first question is: “What does ‘significantly’ mean – that is, what is the variation in each group around these reported means?”
To answer that, I looked at the paper. The results from this study were reported as follows in the paper [1]:
“Infidelity. . . . participants in the prayer for partner condition reported significantly lower infidelity scores (M = 2.44, SD = 1.04) than did those in the neutral condition (M = 3.91, SD = 2.16), F(1, 78) = 7.61, p < .01, eta_p^2 = .87, and the positive thoughts about partner condition (M = 3.90, SD = 2.37), F(1,78) = 6.70, p < .01, \eta_p^2 = .80, but not those in the undirected prayer condition (M = 3.19, SD = 2.11), F(1, 78) = 2.02, p < .16, \eta_p^2 .45. However, those in the undirected prayer condition did not significantly differ from the other two conditions ( ps > .05, ns). All the means reported in this and subsequent analyses were adjusted for the covariates (see Table 2).”
The notation of the statistical analysis used here, ANCOVA, is described elsewhere [3]. The F-value and p-value above are as determined from the standard Fisher Test. However, the key here is that the means have uncertainties – standard deviations, or “SD” – which result from the small statistics of the samples (four groups made from 83 students means roughly 21 students per group, which is not a lot of people per group). If I took four random samples of 21 students from a single group of 83 total students, whose total average scores were 3.5 and whose SD for those scores were about 1-2 overall, I could easy generate four samples with this trend. The reality is that there is no statistically significant difference between these four groups.
To review the science here, 83 undergrads were broken into 4 groups. One group had to pray for their romantic partner each day (using a prescribed prayer), 2 had to conduct undirected prayer, and the last simply had to reflect on the day. All groups performed their assigned activity for the same time, once per day. At the end of the study, the mean infidelity score for the directed prayer group was 2.44 +/- 1.04. For the two undirected prayer groups, it was 3.19 +/- 2.11 and 3.90 +/- 2.37. For the neutral group (daily reflection) it was 3.91 +/- 2.16. Nothing can be concluded from this limited data set – these numbers are, within the rules of statistics, all the same. In order to shed any meaningful light on this question, a sample size at least 3-5 times bigger is needed.
The worst criticism I have of this study is that in its present form it CANNOT be conducted blind. The participants were quizzed on their fidelity before being broken into groups. One of those groups was asked to pray for their partner. Those people could easily be consciously or unconsciously influenced to act more faithful over the 4 weeks of the study, since there is likely some sense that their fidelity is being directly tested. There is no way to control for that in this study, and by construction it is potentially biased. At the very least, that bias would need to be quantified.
It’s my opinion that it was irresponsible for The Economist to fail to report the standard deviations in this study and to simply regurgitate the conclusions of the study without thinking about the results. The Economist is a major, international news analysis of serious record. Regarding the study authors, the fact that the study uses a small sample and doesn’t include any attempt at blinding or bias correction suggests this area of research needs more maturity. Until then, nobody should take it seriously.
When people want aspects of religious belief to enter the realm of scientific scrutiny, two things need to happen. First, those conducting the test owe those of religious faith the best possible scientific practice; this study, in my opinion, falls woefully short of that responsibility. Second, those whose faith is, literally, being put to the test should demand better methods but be prepared to expect results at odds with their belief system. After all, nature does as nature does, no matter what you or I believe. At the very least, inadequate studies like this only serve the confirmation bias, rather than real scientific discovery.
[1] “Faith and unfaithfulness: Can praying for your partner reduce infidelity?” http://www.apa.org/pubs/journals/psp/
[2] http://www.economist.com/node/16886238
[3] ANCOVA: http://udel.edu/~mcdonald/statancova.html
2 thoughts on “Beliefs Deserve Better Science”
At first I interpreted the study to be that the students prayed for their partners and at the end their PARTNERS infidelity was measured. That study could be more amusing, especially if the correlation went the opposite direction.
Unfortunately I’m not having any luck getting the full text of the article. From your description it sounds like they claimed the directed prayer had an effect while the undirected prayer didn’t. What was the text of this directed prayer? I’d find it interesting to see if using a similar text (likely full of positive stuff) in terms of a non-religious daily affirmation would have equivalent/any effect.
The paper is available if your academic institution subscribes to the journal and provides a web proxy that grants access through that subscription. Libraries, such as those at Stanford and SLAC, should offer you access. Here is a direct link to the PDF: http://psycnet.apa.org/psycarticles/2010-16989-001.pdf
I went back and looked more closely over the part of the study that discussed whose fidelity was being judged. The paper seems to make it clear that the participants in the study, who were the ones doing the praying, were also the ones who had to perform the fidelity tests. While reports of behavior were uses to cross-check the results from the fidelity metric, those reports appear to _also_ be generated by questioning the participants. To quote the relevant text from the publication:
“Infidelity (thoughts and acts). To measure infidelity we used a nine-item infidelity scale designed for young adult dating relationships that measures both emotional and physical infidelity (Drigotas et al., 1999) . . .
Participants were instructed to think of a person that they were
most attracted to besides their partner. They were then asked nine questions about their level of attraction (e.g., ‘How attractive did you find this person?’) arousal (e.g., ‘How much arousal did you feel in their presence?’), emotional engagement (e.g., ‘How emotionally intimate were you with this person?’), and physical involvement (e.g., ‘How physically intimate were you with this person?’). The items were summed and coded such that higher scores indicated higher infidelity . . . ”
and
“Infidelity acts. To ensure that praying for partner was not simply leading to report of fewer immoral thoughts, we also looked separately at reports of behavior. These comprised two items from the Drigotas et al. (1999) measure in which participants identified a person for whom they felt attracted. Participants were asked, ‘How emotionally intimate were you with this person?’ and ‘How physically intimate were you with this person?'”
(Emphasis is mine in both quotations)
Here is the prayer used by the group conducting directed prayer for their partner:
“Dear Lord, thank you for all the things that are going well in my life and in
my relationship. Please continue to protect and guide my partner, providing strength and direction every day. I know you are the source of all good things. Please bring those good things to my partner and make me a blessing in my partner’s life. Amen.”
One of the four groups was directed to reflect on the positive aspects of their partner (a positive affirmation).