Bad Science Writing: College Paper Edition

Writers are born young. Good writers learn their craft through practice, trial, and error. Failure is the best teacher. Given my view of writing, there is much my own University’s weekly campus paper, the SMU Campus Weekly, can learn from this recent article that claims to assess the “Paleo diet.” [ARTICLE] I got so upset about this article and its complete lack of scientific assessment, I actually wrote a letter to the Editor of the paper. This is not the first credulous diet review article in our campus newspaper, but it is the last straw for me – a professor who teaches a class in the scientific method and how to construct a good assessment of a claim. This article represents the worst kind of science writing. Check out the article. Then read my letter below. I marked typos that I wish I’d caught before sending the letter with “[sic].”

UPDATE: I updated this after receiving a response from the Editor. I summarize her response, print my own email reply to her, and then document an assessment of the claimed “experts” used in the article.

Dear Editor,

I am writing in regards to the recent piece that appeared in your paper
entitled “Return to your roots with the Paleo diet,” [1] written by Mia
Wennick. I am an Associate Professor of Physics here at SMU, and while
my expertise lies not in Anthropology or Biology or Medicine, I am one
of several instructors that co-teach KNW2333, “Introduction to the
Scientific Method.” In this course, we engage with students in how to
assess new information and to determine, in a scientific sense, whether
or not a claim holds up to scrutiny.

I was thus intrigued by the article mentioned above, especially because
in the Issuu digital print version of the article it is accompanied by a
banner graphic that stated “Debunking Diets.” “Debunking” is defined as
an act meant “to show that something (such as a belief or theory) is not
true : to show the falseness of (a story, idea, statement, etc.);” [2].
Diets, in general, are an area of our lives rife with fad claims that
are quite often unsupported by actual scientific research; thus I was
excited to see what research your staff writer had done to help students
choose better what they do and do not eat.

I was greatly disheartened by nearly every feature of this article.
First and foremost was a lack of a statement to the readers to suggest
that before they embark on any change in their diet that they consult a
medical doctor. Changing diet is a difficult physical process, one
fraught with possible health consequences; it is always wise to consult
a medical doctor before embarking on a major change of diet, especially
if there are medical factors (e.g. a family history of heart disease or
hypertension) that might warrant caution when making major changes in
the ratio of fat and protien and carbohydrate intake.

Beyond the lack of a medical disclaimer to the readers, the article was
far, far from a critical assessment of the Paleo diet; such a critical
assessment is a key feature of “debunking.” Instead, the article was
credulous beyond description – though I will try to address key flaws
below. Rather than using peer-reviewed, journal-published assessments of
the Paleo diet, or even consulting experts in Anthropology and Biology
at SMU (of which there are many!), the article committed the following
most severe research and reasoning fallacies (in addition to many others):

1) Anecdotal Evidence: the weakest form of evidence known to science,
the article relies heavily on testimony from students and never
discussed anything about the reliability of testimony or, more
importantly, the role of the Placebo Effect in personally assessing the
outcome of a treatment or change in lifestyle. The gold standard medical
test is a double-blind, randomized, placebo-controlled study. Has any
such study ever been conducted? Was it replicated? What did it find? We
can’t know from this article. Absent such gold-standard evidence, should
people really switch diets? The staff writer never addressed this [sic]
crucial issues.

2) Supporting evidence came only from the claimant: one of the founding
claimants of the Paleo Diet, Robb Wolf, is selling a book. A journalist
should be skeptical of this person’s claims – they obviously want to
make money off their book. Instead, your staff writer took Wolf’s claims
at face value without assessing the reasons for them. More in-depth
scientific reporting on the Paleo diet has found that it is
fundamentally at odds with the actual understanding of ancient human
diets and human genetics [3][4], something which your staff writer would
have learned if they had consulted SMU Anthropologists and Biologists.

3) Dietitian as Medical Expert: The SMU Dietitian is used as a medical
expert, but Dietitians in general receive little formal medical training
and almost no training is assessing claims scientifically. Dietitians
are only required only to complete a 2 year Masters Program; practicing
scientists are required to complete 4-7 year Ph.D. programs and Medical
Doctors are required to complete about 4 years of intense medical
training. There is a huge difference in basic expertise here. The
Dietitian’s job is largely focused on personalizing to individuals the
current guidelines on healthy eating and lifestyle; they are not
equipped, in general, to assess problems like the Placebo Effect,
medical testing and its reliability, or the modern picture of genetics
and human evolution. While it’s interesting to get their perspective,
they are not scientists nor are they medical doctors. An appeal to
authority is used by stating her credentials after her title (“Courtney
De La Rosa, MCN, RDN, LD”); however, none of those are medical doctor
nor scientific credentials.

4) No disconfirming evidence is presented. Is there truly none? Other
science writers have found it… so why is it not here?

An article like this, submitted as a research outline or paper in
KNW2333, would receive a failing grade – not because I disagree with the
conclusions, but because the methodology… the science writing and the
assessment… is extremely poor. If such articles are to accompany the
word “debunk,” it is important for your writer to actually critically
assess claims using the best scientific information.

Students are busy, nut [sic] they need to make informed choices. This article
parrots the claims of the proponents of the diets, uses weak but popular
forms of “evidence” to support those claims, and then concludes with a
sense of false balance: “Just like any other diet, there are both
believers and critics, but to many the Paleo diet just seems to make
sense.” It’s not about belief. It’s about evidence. A good science
writer knows this. Diets and their impacts can and have been assessed
scientifically. Your writer should know this if they are going to assess
such issues and then offer their insights to our student body.

I hope your will take these criticisms seriously and next time such an
article is up for publication, please consider a few simple questions:

1) Does the evidence for the claim come from anyone other than its
proponents? Are the claimants experts in the area where they make their
claim?

2) Has anyone independently assessed the claim? Are they also experts in
the area of the claim? If so, was gold-standard medical testing used for
the assessment? Was it published in a long-standing, widely regarded,
peer-reviewed journal? Was it replicated?

3) What is the weight of the evidence, and does it merit making a
decision on the part of individuals?

4) Should a medical doctor be consulted first before making any decisions?

Sincerely,
Stephen Sekula
Associate Professor of Physics

[1]
http://www.smudailycampus.com/ae/return-to-your-roots-with-the-paleo-diet

[2] http://www.merriam-webster.com/dictionary/debunk

[3]
http://www.scientificamerican.com/article/why-paleo-diet-half-baked-how-hunter-gatherer-really-eat/

[4]
http://www.theguardian.com/lifeandstyle/2015/aug/18/paleo-diet-critics-science

[ARTICLE] http://www.smudailycampus.com/ae/return-to-your-roots-with-the-paleo-diet

Update #1: Response from the Editor

About 1 day after sending my letter, I received a response from the Editor of the SMU Campus Weekly newspaper. I was glad that she took time to consider the letter and compose a thoughtful response. I don’t want to cut-and-paste her email here, because I didn’t ask her for permission to do so. However, I can try to dutifully summarize the responses and then provide my response email below.

• First and foremost, she was responsible in noting that she can’t speak for the writer. She instead used the letter to attempt to respond to my criticisms from the perspective of an Editor.
• She claimed that she believed the author’s intent was not to encourage readers to change their diets, but rather to explain why the Paleo Diet had become popular.
• The Editor agreed that consulting experts in the faculty at SMU would have added to the article.
• She disagreed with my argument that appropriate expertise had not been used. She noted that the article writer had used information from the diet’s primary claimant, Robb Wolf, a biochemist, and that the article writer had used information from a .com website used as a central hub for information about the diet. She argued that Wolf, and the website’s owner, Loren Cordain (a Ph.D.), are sufficient expertise.
• She disagreed with my criticism of the qualifications of the SMU dietitian as an expert for this article. She argued that the dietitians viewpoints are still valid, even if she has less training than a scientist or a medical doctor.
• She said the author did not a flaw in the diet: its lack of carbohydrates.
• She apologized if the article banner, “Debunking Diets,” was misleading as to the purpose of the article.
• She thanked me for my letter and said that she would explain my points to writers at the newspaper interested in writing on similar topics.

So there is a lot to be encouraged and disheartened about in this reply from the Editor.

Here was my email in response to her reply:

Thank-you very much for your thoughtful and prompt response. I appreciate the feedback on my own letter.

Let me follow-up on just two of the points you make in your response email. One of the things we cover in our course, as all similar courses do, is reliability of sources. Of course, this is a the heart of journalism. Anyone can setup a .com website; they cost just $5-$15 per year to own and run. Publishing a book is fairly easy, too, all things considered. Doing these things requires no credentials. A journalist should always be wary of information that comes from any website that is not .gov or .edu because there is no review required for information that appears on other domains. It’s always wise to find a peer-reviewed, published source. In addition, it is also always prudent to seek sources of information from people _other_ than the claimants. They clearly have a financial motive, even if they truly believe in what they say. But belief is irrelevant in decisions where the claimants state that facts back them up; all that then matters is the quality and amount of evidence for or against the claim, and here there is a mountain of evidence against the claims of the Paleo Diet.

In addition, being a biochemist (as Wolf states to be his area of expertise) does not make one an expert in the Anthropology of human diets or the way in which genetics and diet interact, or even in medicine and medical testing. This person is making claims that fall outside the bounds of their expertise. Luckily, there are many experts in these fields that have looked at the claims of the diet and find they don’t hold up under any scientific scrutiny.

If you’d like to meet and talk further, of course I am happy to do so. I’m sure you’re busy, but this subject is one about which I am passionate (the process needed to make reliable decisions using information) and I am always pleased to speak in person. Email is impersonal and I find it too easy to miscommunicate. However, I am very satisfied that you took the time to respond.

Sincerely,
Stephen Sekula

Who are our two big “experts”?

Who are Robb Wolf and Loren Cordain? These are the credentialed people touted as being reliable sources of information about the very diet they advocate for (already, mixing scientific claims with financially motivated advocacy is a huge problem… but I digress).

A quick Google search for “Rob Wolf paleo diet” turns up, of course, the most popular or highly linked sites first: Rob Wolf’s personal website , for instance, and tons of websites selling his book. On his own website, in what should be his own words, Wolf describes himself as follows:

ROBB WOLF, author of The Paleo Solution, is a former research biochemist and one of the world’s leading experts in Paleolithic nutrition. Wolf has transformed the lives of tens of thousands of people around the world via his top ranked iTunes podcast and wildly popular seminar series. [ROBBWOLF]

And on the “About” page from he personal website, we learn a very interesting thing from him:

Robb Wolf, a former research biochemist is the New York Times Best Selling author of The Paleo Solution – The Original Human Diet. A student of Prof. Loren Cordain, author of The Paleo Diet, Robb has transformed the lives of hundreds of thousands of people around the world via his top ranked iTunes podcast, book and seminars.

Robb has functioned as a review editor for the Journal of Nutrition and Metabolism, is co-founder of the nutrition and athletic training journal, The Performance Menu, co-owner of NorCal Strength & Conditioning, one of the Men’s Health “top 30 gyms in America” and he is a consultant for the Naval Special Warfare Resiliency program. He serves on the board of Directors/Advisors for: Specialty Health Inc, Paleo FX, and Paleo Magazine. [ROBBWOLF]

So we have already learned something useful from Mr. Wolf, even from his own self-description: he is a FORMER research biochemist. He was a student of the second “expert” used by the SMU Campus Daily expert, Loren Cordain. He also claims to have been a review editor for “Journal of Nutrition and Metabolism” (thought I can find no mention of him anywhere on the journal publisher’s website – certainly not on the Editorial board of this journal).

But where has be been a “research biochemist”? How did he earn this title? What degrees does he hold? What research has he published in peer-reviewed journals? And how does this make him “…one of the world’s leading experts in Paleolithic nutrition”?

I went to answer the “peer-reviewed publications” questions first; after all, output is a hallmark of the quality of a scientist and their methods (use of that output is just as important, if not more). Wolf has no CV. So, I searched in the SMU Central University Library system for “Scholarly and Peer Reviewed” “Journal Articles” whose author is “Robb Wolf”. I found 34 articles that matched my search query. However, NONE of them were by our Robb Wolf; most were hits based on false matches to my search query (e.g. Wolf and Robb appear in the author list, but not as the name of a single individual). So Wolf appears to have NEVER published anything that stood up to peer-review. That is a huge red flag. How can he be an expert in anything if nothing he has done is disseminated and recognized in the scientific community? It took me 10 minutes to establish this.

Where did he earn his degrees, and what degrees does he hold? This was a little harder to track down. I decided to start with his claimed mentor, Loren Cordain [LORENCORDAIN]. Presumably, Wolf earned degrees from whatever institution Cordain was at.

Cordain was, until 2013. a Professor in the Colorado State University Department of Health and Exercise Science. According to his self-described biography,

Dr. Loren Cordain received his Ph.D. in Health from the University of Utah in 1981, his master’s in physical education from the University of Nevada-Reno in 1978, and his bachelor’s in health sciences from Pacific University in 1974. He was a professor in the Department of Health and Exercise Science at Colorado State University from 1982 – 2013. [LORENCORDAIN]

This all seems pretty above-board… but how does a person with a Ph.D. in “Health” and a Masters Degree in Physical Education become the self-described “…world’s foremost authority on the evolutionary basis of diet and disease”? He claims to have published “…more than 100 peer-reviewed scientific articles and abstracts, and his research into the health benefits of Stone Age Diets for contemporary people has appeared in the world’s top scientific journals including the American Journal of Clinical Nutrition, the British Journal of Nutrition, and the European Journal of Clinical Nutrition, among others.” [LORENCORDAIN].

Let’s look at his publication record, from his own CV [CORDAINCV]. First, abstracts and conference proceedings require zero peer review. I ignore those. You can pretty much say anything you want in an abstract or conference talk and never have to be held accountable by the community. I focused on his peer-reviewed, refereed journal publications, of which he claims 90. The first third or so of his papers have to do with studies of athletics under different conditions – comparing men and women, or testing some new piece of athletic equipment. Here are the papers that are most relevant to his claimed area of expertise (I ignore “letters” top the journal – again, not refereed or peer-reviewed):

• Cordain, L., Gotshall, R.W., Eaton, S.B. (1997). Evolutionary aspects of exercise. World Review of Nutrition and Dietetics, 81: 49-60.
• Eaton, S.B., Cordain, L. (1997). Old genes, new fuels: Nutritional changes since agriculture. World Review of Nutrition and Dietetics, 81: 26-37.
• Cordain, L., Gotshall, R.W. and Eaton, S.B. (1998). Physical activity, energy expenditure and fitness: an evolutionary perspective. International Journal of Sports Medicine, 19(5): 328-335.
• Eaton SB, Eaton SB Jr, Cordain L, Mann N, Sinclair A. (1998). Dietary intake of long chain polyunsaturated fatty acids during the paleolithic. World Review of Nutrition and Dietetics, 83: 12-23.
• Cordain L. (1999). Cereal grains: humanity’s double edged sword. World Review of Nutrition and Dietetics, 84: 19-73.
• Powell PD, DeMartini JC, Azari P, Stargell LA, Cordain L, Tucker A. (2000). Evolutionary stable strategy: A test for theories of retroviral pathology which are based upon the concept of molecular mimicry. Journal of Theoretical Biology, 202:213-229.
• Cordain L, Brand Miller J, Eaton SB, Mann N, Holt SHA, Speth JD. Plant to animal subsistence ratios
  and macronutrient energy estimations in world wide hunter-gatherer diets. American Journal of Clinical
  Nutrition, 2000, 71:682-92.
• Cordain, L., Brand Miller, J., Eaton, S.B. & Mann, N. (2000). Hunter-gatherer diets – a shore based
  perspective (letter). American Journal of Clinical Nutrition, 72: 1585-86.
• Cordain, L., Brand Miller, J., Eaton, S.B. & Mann, N. (2000). Macronutrient estimations in hunter-
  gatherer diets (letter). American Journal of Clinical Nutrition, 72: 1589-90.
• Cordain L, Watkins BA, Mann NJ. Fatty acid composition and energy density of foods available
  to African hominids: evolutionary implications for human brain development. World Review of Nutrition
  and Dietetics, 2001, 90:144-161.
• Eaton SB, Cordain L, Eaton SB. An evolutionary foundation for health promotion. World Rev Nutr Diet
  2001; 90: 5-12.
• Eaton SB, Strassman BI, Nesse RM, Neel JV, Ewald PW, Williams GC, Weder AB, Eaton SB 3rd, Lindeberg S, Konner MJ, Mysterud I, Cordain L. Evolutionary health promotion. Prev Med 2002;34:109-118.
• Eaton SB, Cordain L. Evolutionary Health Promotion. A consideration of common counter-arguments. Prev Med 2002;34:119-123.
• Cordain L. Book Review. Hunter-Gatherers: An interdisciplinary perspective. Am J Hum Biol 2002;14: 280-281.
• Cordain L, Watkins BA, Florant GL, Kehler M, Rogers L, Li Y. Fatty acid analysis of wild ruminant tissues: Evolutionary implications for reducing diet-related chronic disease. Eur J Clin Nutr, 2002;56:181-191.
• Cordain L, Eaton SB, Brand Miller J, Mann N, Hill K. The paradoxical nature of hunter-gatherer diets: Meat based, yet non-atherogenic. Eur J Clin Nutr 2002;56 (suppl 1):S42-S52.
• Cordain L, Eaton SB, Brand Miller J, Lindeberg S, Jensen C. An evolutionary analysis of the etiology and pathogenesis of juvenile-onset myopia. Acta Opthalmolgica, 2002,80:125-135.
• Cordain L. The nutritional characteristics of a contemporary diet based upon Paleolithic food groups. J Am Neutraceut Assoc 2002; 5:15-24.
• Cordain L, Lindeberg S, Hurtado M, Hill K, Eaton SB, Brand-Miller J. (2002). Acne vulgaris: A disease of civilization. Archives of Dermatology,138: 1584-90.
•  Cordain L, Eades MR, Eades MD. (2003). Hyperinsulinemic diseases of civilization: more than just syndrome X. Comp Biochem Physiol Part A:136:95-112.
• Cordain L. Response to omega-3 fatty acids and acne. Arch Dermatol 2003;139:942-3.
• Lindeberg S, Cordain L, Eaton B. Biological and clinical potential of a Palaeolithic diet. J Nutr Environ
  Med 2003;13:149-160.
• O’Keefe J.H., Cordain L. Cardiovascular disease as a result of a diet and lifestyle at odds with our Paleolithic genome: how to become a 21st century hunter-gatherer. Mayo Clin Proc 2004;79:101-108.
• Cordain L, O’Keefe J.H. Reply: The hunter-gatherer diet. Mayo Clin Proc 2004; 79:703-04.
• Abuissa H, O’Keefe JH, Cordain, L. Realigning our 21st century diet and lifestyle with our hunter-gatherer genetic identity. Directions Psych, 2005;25:SR1-SR10.
• Cordain L., Hickey MS. Ultraviolet radiation represents an evolutionary selective pressure for
  the south-to-north gradient of the MTHFR 677TT genotype. Am J Clin Nutr, 2006;84:1243.
• Eaton SB, Cordain L, Sparling PB, Cantwell JD. Evolution, body composition and insulin resistance. Preventive Medicine, 2009;49:283-285.
• Ramsden CE, Faurot KR, Carrera-Bastos, P, Sperling LS, de Lorgeril M, Cordain L. Dietary fat quality and coronary heart disease prevention: a unified theory based on evolutionary, historical, global and modern perspectives. Curr Treat Options Cardiovasc Med, 2009;11:289-301.
• Eaton SB, Konner MJ, Cordain L. Diet-dependent acid load, Paleolithic nutrition, and evolutionary health promotion. Am J Clin Nutr, 2010;91:295-97.
• O’Keefe JH, Vogel R, Lavie CJ, Cordain L. Achieving hunter gatherer fitness in the 21st century. Am J Med, 2010 Sep 13. [Epub ahead of print]
• Remko S. Kuipers1, Martine F. Luxwolda1, D.A. Janneke Dijck-Brouwer1, S. Boyd Eaton, Michael A. Crawford, Cordain L, and Frits A.J. Muskiet. Estimated macronutrient and fatty acid intakes from an East African Paleolithic diet. Brit J Nutr, 2010 Dec;104(11):1666-87.
• O’Keefe JH, Vogel R, Lavie CJ, Cordain L. Organic Fitness: Physical Activity Patterns Compatible with our Hunter Gatherer Genetic Legacy. Physician and Sports Medicine, 2010, 38 (4):11-18.
• Carrera-Bastos P, Fontes Villalba M, O’Keefe JH, Lindeberg S, Cordain L. The western diet and lifestyle and diseases of civilization. Res Rep Clin Cardiol 2011; 2: 215-235.
• O’Keefe JH, Vogel R, Lavie CJ, Cordain L. Exercise Like a Hunter Gatherer: A Prescription for Organic Physical Fitness. Prog Cardiovasc Dis. 2011;53:471-9.
• Fontes-Villalba M, Carrera-Bastos P, Cordain L. African hominin stable isotopic data do not necessarily indicate grass consumption. Proc Natl Acad Sci U S A. 2013 Oct 22;110(43):E4055. doi: 10.1073/pnas.1311461110. Epub 2013 Sep 23.

It’s impossible for me to go through the entire history of this person’s relevant publication list and assess it. Luckily, others have (you’ll note a lot of these are responses to criticism of previous articles on subjects, and there are more such in the less-relevant publications in his CV). This suggests people are refuting and assessing Cordain and his colleagues’ claims, and that this is a lively area of actual scientific debate. That’s a good sign. It doesn’t mean we should accept or reject his ideas, per se; it only means that people are doing their own assessments of his claims.

But… one thing I noted in the entirety of Cordain’s CV was this: there is no mention of Robb Wolf anywhere. Robb is not listed as a Ph.D. or Masters student under Cordain. So just who is this Robb Wolf?

It’s completely unclear. A Wikipedia page that never appeared on Wikipedia but is archived [WIKIPEDIAWOLF] claims, without reference to a source, “He earned a BS in Biochemistry from California State University Chico. He then worked as a research biochemist for 5 years, which included lipid metabolism research at the Fred Hutchinson Cancer Research Center. He also was a research assistant with Prof. Loren Cordain of Colorado State University.”  The best I could find otherwise was a sparse LinkedIn page [LINKEDINWOLF], noting the B.S. in Biochemistry from Cal-State Chico but zero mention of this Cancer Center or how he ever worked with Cordain.

So he is a biochemist in the same sense that an SMU graduate with a B.S. in Political Science, who then does 5 years of clerking at a law firm, is a Supreme Court Justice. Sure, he has the basic training and coursework that you’d WANT in a scientist, but his experience after college is unclear and thin at best – 5 years of lab work is barely enough to qualify for starting a Ph.D. thesis. He might have been a research assistant with Cordain, but Cordain never graduated him with an advanced degree. And if all of the above is true, how does a B.S. in biochemistry from Cal-State Chico and 5 years of research on “lipid metabolism,” qualify ANYONE to be the “world’s leading expert on Paleolithic nutrition”? Wolf is an unreliable expert AT BEST, and since little of his expertise can be corroborated with public sources a journalist should just ignore him. He’s clearly selling an image and a book, all at the same time.

So of the two big “experts” cited in the SMU Campus Weekly article – Wolf and Cordain – one of them is sketchy at best as regards scientific credentials (Wolf), while one was a legit scientist but his work has been widely refusted as an inaccurate picture of human biology and also ancestral diets. It would have been good to bring a contemporary, local, third expert into this – a faculty member from SMU or UT-Southwestern who is truly an active expert in this area.

What about that criticism of the Dietitian?

As for the use of the Dietitian as the sole local SMU expert, I am sad to say in response to the Editor that she is NOT as expert as someone with a medical or research degree. It’s not true that the opinion of a person with 2 years of training should be counted equally with that or someone with at least 5-7 years of medical or scientific training and then decades of experience in the field afterward. In science, all opinions are not counted equally. A person with experience in Anthropology, Biology, or Medicine would count a LOT more than that of a university dietitian.

So I stand by my own criticisms of this article. It was poorly researched and, at best, credulous.

[ROBBWOLF]: http://robbwolf.com/

[LORENCORDAIN]: http://thepaleodiet.com/dr-loren-cordain/

[CORDAINCV]: http://thepaleodiet.com/wp-content/uploads/2015/09/Loren-Cordain-CV-9-2015-1.pdf

[WIKIPEDIAWOLF]: https://en.wikipedia.org/wiki/User:Kenmcl2/Robb_Wolf

[LINKEDINWOLF]: https://www.linkedin.com/in/robb-wolf-008b996a