NPR Science Fail: diet disclaimer

NPR’s science reporting on diet and nutrition has lately been dismal. Graphic by Leo Shaw (Ref. 2).

During my listening to NPR over the past 4 days, I have heard some major science reporting fails. I reported on one the other day: the way in which a study of organic food nutrition was reported. Today, while working out in the early hours of the morning, I heard a second.

NPR’s “Morning Edition” did a two-story series related to the diabetes epidemic accompanying the obesity epidemic in the U.S. The first story in the series focused on diet [1]. Specifically, it focused on the claim that some diets are better than others when it comes to weight loss and/or weight maintenance. The story VERY SPECIFICALLY focused on research conducted by one medical researcher, David Ludwig (who holds both a Ph.D. and M.D.). There is nothing, per se, wrong with the research. However, there was something fundamentally wrong with the reporting.

This is how the story SHOULD have gone:

” . . . Ludwig and colleagues recently published a study in the Journal of the American Medical Association that offers some tools you might use to fight back. Researchers compared the low-carb, low-fat and low-glycemic diets to see which one burned the most calories per day. The low-carb diet was the clear winner. The low-fat diet was the loser. But it was the diet in the middle, the low-glycemic index diet, that Ludwig suggests is more promising. It burned more calories per day than the low-fat diet and proved easier to stick to over the long term than the low-carb diet. Ludwig is quick to caution that his study was short and not conclusive. He’s working now to design a long-term study that looks at diet and weight loss maintenance over a number of years.”

This, unfortunately, was how the story was assembled:

” . . . Ludwig and colleagues recently published a study in the Journal of the American Medical Association that offers some tools you might use to fight back. Researchers compared the low-carb, low-fat and low-glycemic diets to see which one burned the most calories per day. The low-carb diet was the clear winner. The low-fat diet was the loser. But it was the diet in the middle, the low-glycemic index diet, that Ludwig suggests is more promising. It burned more calories per day than the low-fat diet and proved easier to stick to over the long term than the low-carb diet.

… [FOUR PARAGRAPHS OF WRITTEN MATERIAL, OR ABOUT 3 MINUTES INTO A 4:15 MINUTE STORY AND AFTER AT LEAST 1 ANECDOTE] …

. . . Ludwig is quick to caution that his study was short and not conclusive. He’s working now to design a long-term study that looks at diet and weight loss maintenance over a number of years.”

So while Ludwig was quick to caution the reporter about the inconclusiveness of his own work, the reporter (Patti Neighmond) was NOT quick to report on that inconclusiveness. Until the last 55 seconds of the story, in fact, the reporter is completely credulous about the findings of Ludwig and his colleagues! It’s only at the very end that she brings in a dietitian to talk about the importance of person-to-person variation in factors related to weight loss and weight maintenance, and the importance of EXERCISE to those two goals.

Of the three big diet stories I’ve heard recently aired on either NPR’s “Morning Edition” or “All Things Considered”, two of them have been highly questionable in the way in which the science was presently.

Usually, NPR does a fantastic job of reporting on science. Even with these fails, they are still doing a better job than the network or cable news outlets. I still recommend NPR science reporting to the casual listener; but, I expect better from the best.

 

[1] http://www.npr.org/blogs/thesalt/2012/09/10/160757730/low-and-slow-may-be-the-way-to-go-when-it-comes-to-dieting

[2] http://eurofail.blogspot.com/2009/01/science-fail.html

Reporting on the health benefits of organic food

When Michael Pollan described in “The Omnivore’s Dilemma” some of the impacts of industrial scale nitrogen-fertilizer farming, he wrote passionately about the Des Moines river in Iowa:

But what happens to the one hundred pounds of synthetic nitrogen that Naylor’s corn plants don’t take up? Some of it evaporates into the air, where it acidifies the rain and contributes to global warming. (Ammonium nitrate is transformed into nitrous oxide, an important greenhouse gas.) Some seeps down to the water table. When I went to pour myself a glass of water in the Naylors’ kitchen, Peggy made sure I drew it from a special faucet connected to a reverse-osmosis filtration system in the basement. As for the rest of the excess nitrogen, the spring rains wash it off Naylor’s fields, carrying it into drainage ditches that eventually spill into the Raccoon River. From there it flows into the Des Moines River, down to the city of Des Moines—which drinks from the Des Moines River. In spring, when nitrogen runoff is at its heaviest, the city issues “blue baby alerts,” warning parents it’s unsafe to give children water from the tap. The nitrates in the water bind to hemoglobin, compromising the blood’s ability to carry oxygen to the brain. So I guess I was wrong to suggest we don’t sip fossil fuels directly; sometimes we do. [1]

There is no credible evidence that conventional and organic food differ in their direct health benefits to people. How is the press reporting on this recent conclusion from a meta-study of peer-reviewed publications on the subject? And does that reaction encapsulate the many dimension of this issue? Photo from Ref. 8.

Many such farms run off into rivers. Farms that run off into the Mississippi have altered the balance of life in the Gulf of Mexico. The nitrogen-enriched water provides nourishment to algaes, whose populations explode in the Gulf and leech oxygen out of the water. This results in “dead zones,” places where oxygen depletion is so severe that oxygen-dependent  marine life cannot exist there.

“Sustainable farming” is a practice of planting and harvesting crops using methods and cycles that attempt to avoid altering severely the local ecology; another aspect of such practice is to take advantage of such ecology to enhance the pollination and growth cycles. What is being sustained is the existing balance of life. This is meant to be in contrast to the kinds of effects described by Pollan or seen in the Gulf, where the balance of life is altered to favor new species at the expense of the existing dominant ones.

“Organic farming” is often conflated with sustainable farming, though they are not necessarily the same thing. “Organic” is compromise label earned by farmers from the US Department of Agriculture, and while it can be used to label products raised according to a set of standards those standards are not necessarily the same as those employed in sustainable farming. Often, “organic” is used to imply not only the means by which fruits and vegetables are raised, fed, and protected, but also applied to the means by which livestock are also raised, fed, and protected. The use of classes of chemical pesticides, for instance, is banned if you wish to label food “organic.”

But in parallel with this organic food labeling boom, a lot of other claims exploded regarding the health benefits or impacts of organic food. For instance,

  • because organic food is exposed to less (or no) chemical pesticides, it’s better for your health (e.g. leads to a lower incidence of cancer or other negative health impacts)
  • because organic food is raised without the aid of chemical pesticides or fertilizer, its nutrient content is higher and, if more nutrients are better for health (also a debatable issue), then organic food is better for you

None of these claims have anything to do with the issues at the heart of sustainable farming, where the focus is not on making more nutritious food but instead is on respecting the ecology of the surrounding environment and taking advantage of it to help produce the food without significant negative impacts (e.g. those that alter the balance of the ecology).

The health claims have been very popular. Dozens of websites appears on the first page of a Google search for “nutrition benefits of organic food,” and many of those sites use questionable research to back up their claims. For instance, the logically-fallacy-named “Organic Authority” (Appeal to Authority)  says [2] (emphasis and formatting are theirs),

But don’t just take it from me…other vetted resources agree that organic food is nutrient-rich and fantastic for your health…

Produce Producing Nutrients!

In a study published in March 2008 by The Organic Center, scientific evidence settled the lingering question: “Are organic foods really more nutritious?”

The answer is a resounding “Yes!” Consider the following:

  • Organic plant-based foods (fruits, vegetables, grains) contain higher levels of eight of 11 nutrients studied, including significantly greater concentrations of health-promoting polyphenols and antioxidants.
  • Organically grown plant-based foods are 25% more nutrient dense, on average, than their conventional food counterparts. That means they deliver more essential nutrients per serving or calorie consumed than conventionally-grown foods.
  • Nutrients present in organic foods are “in a more biologically active form,” according to Neal Davies, a professor at Washington State University (WSU) and a co-author of the center’s report.

This sounds very fancy . . . until you look at the peer-reviewed scientific research on the question of organic food and nutrition. By the way, the study above from “The Organic Center” (your BS detector should already be buzzing when the research institution is named after the central player in the claim) can be found in Ref. [3] and is an UNPUBLISHED, NON PEER-REVIEWED internal report of the center. Without peer review, there is no way of knowing what shenanigans they engaged in to cherry pick the research they reviewed.

So what DOES the actual published, peer-reviewed literature say? Researchers at Stanford University funded by Stanford’s Medical School assessed the health claims using a number of independent research studies. This is called a “meta-analysis” – an independent assessment of existing literature and a determination of the weight of the evidence. The analysis was published in the Annals of Internal Medicine [3] and is entitled, “Are organic foods safer or healthier than conventional alternatives?: a systematic review.” The authors of the study are Crystal Smith-Spangler, MD, MS; Margaret L. Brandeau, PhD; Grace E. Hunter, BA; J. Clay Bavinger, BA; Maren Pearson, BS; Paul J. Eschbach; Vandana Sundaram, MPH; Hau Liu, MD, MS, MBA, MPH; Patricia Schirmer, MD; Christopher Stave, MLS; Ingram Olkin, PhD; and Dena M. Bravata, MD, MS. [5]

You can read the summary of their findings online, which I reproduce here:

17 studies in humans and 223 studies of nutrient and contaminant levels in foods met inclusion criteria. Only 3 of the human studies examined clinical outcomes, finding no significant differences between populations by food type for allergic outcomes (eczema, wheeze, atopic sensitization) or symptomatic Campylobacter infection. Two studies reported significantly lower urinary pesticide levels among children consuming organic versus conventional diets, but studies of biomarker and nutrient levels in serum, urine, breast milk, and semen in adults did not identify clinically meaningful differences. All estimates of differences in nutrient and contaminant levels in foods were highly heterogeneous except for the estimate for phosphorus; phosphorus levels were significantly higher than in conventional produce, although this difference is not clinically significant. The risk for contamination with detectable pesticide residues was lower among organic than conventional produce (risk difference, 30% [CI, −37% to −23%]), but differences in risk for exceeding maximum allowed limits were small. Escherichia coli contamination risk did not differ between organic and conventional produce. Bacterial contamination of retail chicken and pork was common but unrelated to farming method. However, the risk for isolating bacteria resistant to 3 or more antibiotics was higher in conventional than in organic chicken and pork (risk difference, 33% [CI, 21% to 45%]).

So, in short:

  • All nutrient assessments showed organic and conventional food are the same
  • There are detectable differences between organic food and conventional food regarding phosphorous and pesticide exposure in humans, but those differences are not clinically significant (e.g. they have no measurable health impacts)
  • You are more likely to find antibiotic resistant bacteria in conventionally produced food

The authors then rightly and responsibly conclude:

The published literature lacks strong evidence that organic foods are significantly more nutritious than conventional foods. Consumption of organic foods may reduce exposure to pesticide residues and antibiotic-resistant bacteria.

But these reductions have no medical impacts. Therefore, for all the nutrient and pesticide assessments, it makes no difference whether you consume organic or conventional food. Any claims to the contrary fly in the face of existing and reliable scientific evidence.

This is not to say that more evidence is not needed. More studied would help a meta-analysis improve their understanding of the effects of small statistics (e.g. only 3 human studies assessed clinical outcomes) and bias in methodology. The authors note this in their findings. More science would be better. But anyone making decisions on current science has no basis on which to claim organic food is more nutritious than conventional food.

The press has been widely reporting on this story. I was particularly affected by the reporting of NPR commentator Steve Inskeep and NPR science correspondent Allison Aubrey [6] in that I noted in their language some of the common misunderstandings of science itself. Science is a process. It is not a collection of findings. If the process yields no evidence for the claim (a “null result”), that is, in and of itself, a scientific result. A finding of a lack of evidence does NOT mean that science has not yet been able to find the reason to uphold the original claim; it has determined that the claim has no basis in reality, and cannot be relied upon. More research may one day find evidence for the claim, but if you have to make policy now then you better use what you know, and not what you do not know.

The exchange between Inskeep and Aubrey that concerned me was as follows:

INSKEEP: . . . But there is another question here about the actual findings of this study, saying they can’t find evidence of health benefits to eating organic food. James Johnson of Windsor, California, writes: I honestly can’t believe that a scientist in this day and age would make the claim that pesticides building up in the human body does not matter.

And it certainly does seem intuitive; that it’s better not to be ingesting pesticides than to be ingesting pesticides, Allison.

AUBREY: Well, I think this goes back to the answer I started with; that if this is the question that people want answered, science has not answered it yet. We can make lots of assumptions, but if you look at the body of evidence that these researchers were reviewing, they’re mostly short-term, narrow studies.

For instance, they’re looking at – if pregnant women and their children eat organic, are they going to be less inclined to have allergic conditions; say, eczema? And when they look at that body of evidence, they don’t see any big signs of big benefits. But as the researchers told me – that some of the findings are what they call hypothesis-generating; meaning, they look at the data, and they see a subgroup of people for whom maybe there’s an influence between these two things. So…

INSKEEP: Meaning that maybe in some cases, there is some possibility of an organic benefit. But in studies lasting several years, they have yet to uncover it for sure.

AUBREY: That’s exactly right.

Inskeep and Aubrey engage in classic logical fallacies and popular misunderstandings of science throughout this exchange. Inskeep reads from a listener comment and then injects that ” . . . it certainly does seem intuitive; that it’s better not to be ingesting pesticides than to be ingesting pesticides . . . ” This is the classic “everybody knows” argument, coupled with a misunderstanding of the concept that “the dose makes the poison.” For instance, water in high doses is lethal, inducing a condition known as hyponatremia (this condition once killed an SMU student during a fraternity hazing, and kills 1-2 marathoners per year in the U.S.). Is water, therefore, a poison or not? Well, it depends on the dose. Obviously, you don’t want to eat a whole cup of pesticide. But are trace amounts a problem? According to research, no. So Inskeep is being very irresponsible here.

The reader’s comment also is laced with an unfounded assumption: that by consuming conventional foods with slightly more pesticide content, the pesticides “build up” in the body; Inskeep and Aubrey let that slide, although since that is a founding assumption of the listener’s comment, it ought to be challenged in the response. What is the evidence that pesticides build up over time in the body?

Aubrey is not helping. She executes the mistake of interpreting a null finding as meaning that ” . . . science has not answered [the question] yet . . . ” and, when she agrees later with Inskeep, of implying that because the findings are null that ” . . . maybe there is some . . . organic benefit . . .” but that they have not yet uncovered it.

This is an appeal to wishful thinking. We all wish there were magical secret benefits of some food, but if the science says there is no evidence of that then the only rational conclusion is to say it doesn’t have that benefit. Arguing otherwise is engaging in fantasy-based medicine and magical thinking, neither of which make good medicine or good health policy.

The reality is that proper farming methods have potentially a lot of other benefits, and focusing narrowly on one or the other is chasing the tail of the animal without considering the whole animal. These meta-studies do not address the ecological or carbon emission outcomes of sustainable or organic farming. That’s a totally separate question, linked to human health but also to a number of other issues. One should try to think about the big picture, and do a whole assessment of such farming before singing its praises or showering it with scorn.

What does Michael Pollan have to say about any of this? Despite being a person regarded as a modern figurehead in the organic and sustainable movement, his take is somewhat more measured and reasoned [7] than either Inskeep or Aubrey:

JON BROOKS: So is this meta-study a big deal?

MICHAEL POLLAN: I’m not sure it’s a big deal. The media’s playing it as if there were something new here, but this is not new research, it’s a meta-study [a review of previously conducted research], and I’ve seen the exact same data analyzed in a very different direction. A lot of it depends on how you manage your assumptions and statistical method.

I think we’re kind of erecting a straw man and then knocking it down, the straw man being that the whole point of organic food is that it’s more nutritious. The whole point of organic food is that it’s more environmentally sustainable. That’s the stronger and easier case to make.

It’s true the body of research around nutrition is really equivocal, and we need to do more studies on that. But the success of organic doesn’t stand or fall on that question. This study disputes how significant the differences in antioxidant and nutrient levels are between organic and conventional food. But that’s not central to the discussion of why organic is important, which has a lot more to do with how the soil is managed and the exposure to pesticides, not just in the eater’s diet but to the farmworker.

His point about “I’ve seen the exact same data analyzed in a very different direction” is a logical fallacy; Pollan is not qualified to assess the quality of scientific research, nor of a meta-analysis because he holds degrees in English, not science, and has no research training whatsoever. His opinion is at best an opinion when it comes to the quality or interpretation of medical research data; I put my trust in a dozen medical research scientists from Stanford over that of an English professor and writer. But his larger point – that there is a BIGGER set of issues here – is in the right direction.

Whatever you take away from the Stanford study, it should not be that organic or sustainable farming is not important. It should merely be that food is food, that vigilance over the food production chain produces good quality food with little human consumption downsides from fertilizers or pesticides, and that you can spend your dollar on organic or conventional and get the same nutrition.

[1] Pollan, Michael. “The Omnivore’s Dilemma.” pp. 46-47.

[2] http://www.organicauthority.com/organic-food/organic-food-articles/health-benefits-organic-food.html#s.abqxjxrzqbiaa

[3] The Organic Center, March, 2008 Report: http://www.organic-center.org/science.latest.php?action=view&report_id=126

[4] http://annals.org/journal.aspx

[5] “Are Organic Foods Safer or Healthier Than Conventional Alternatives?: A Systematic Review”
Crystal Smith-Spangler, MD, MS; Margaret L. Brandeau, PhD; Grace E. Hunter, BA; J. Clay Bavinger, BA; Maren Pearson, BS; Paul J. Eschbach; Vandana Sundaram, MPH; Hau Liu, MD, MS, MBA, MPH; Patricia Schirmer, MD; Christopher Stave, MLS; Ingram Olkin, PhD; and Dena M. Bravata, MD, MS. Ann Intern Med. 4 September 2012;157(5):348-366 http://annals.org/article.aspx?articleid=1355685

[6] http://www.npr.org/templates/transcript/transcript.php?storyId=160681396

[7] http://blogs.kqed.org/newsfix/2012/09/04/michael-pollan-organic-study/

[8] http://www.flickr.com/photos/spine/558333048/

Political Courage and Cowardice

When it comes to policy, both U.S. political parties may have their heads in the sand when dealing with climate disruption. But one party has the courage to talk about it, and the other is taking advantage of drought and drier, looser sand to dig their heads in deeper.

You may or may not like Mitt Romney. You may or may not like Barack Obama. But these past two weeks, despite all the scripted speeches and propaganda sloganeering of the Republican and Democratic National Conventions, one of these two demonstrated political courage and the other did not.

The scientific case for human-induced global climate disruption is overwhelming. Well over 95% of active researching and publishing climate scientists agree on this fact and the causes. The public and policy makers are still struggling to figure out howe to incorporate the science into social and economic decision-making. That’s the only place where there is actual controversy. It takes political courage, in the current American climate (pun intended), to even mention climate disruption (also called popularly “global warming” or “climate change”). It takes even more political courage to speak at length about it.

The mention of climate disruption, or even climate science, was almost entirely missing from the Republican platform for 2012. As I noted in a previous post, the only mention of something related to climate disruption had to do with a criticism of Barack Obama and how often he says that word in relation to words related to terrorism. There was complete avoidance of the subject in the platform, and complete avoidance of the subject by Mitt Romney in his acceptance speech [1]. From a scientist’s perspective, that is total political cowardice. For an issue that so clearly presents challenged to the economy and social outcomes of the People of the United States, utter avoidance of the subject is tantamount to grabbing a second shovel to bury their heads even deeper in the sand. I prefer crazy anti-science statements setting up scientists as flammable straw figures easily burned to the ground over total silence on an issue so important to the United States and to the world.

Why did President Barack Obama demonstrate political courage? Because he put this smack in his convention speech [2]:

We’re offering a better path where we — a future where we keep investing in wind and solar and clean coal, where farmers and scientists harness new biofuels to power our cars and trucks, where construction workers build homes and factories that waste less energy, where — where we develop a hundred-year supply of natural gas that’s right beneath our feet. If you choose this path, we can cut our oil imports in half by 2020 and support more than 600,000 new jobs in natural gas alone. (Cheers, applause.)

And yes, my plan will continue to reduce the carbon pollution that is heating our planet, because climate change is not a hoax. More droughts and floods and wildfires are not a joke. (Cheers, applause.) They are a threat to our children’s future.

Despite this mention of such an important scientific issue with deep ties to our species continued success, it’s bittersweet. There’s been no more progress on implementing a national carbon trading/curtailing market that could then be applied as a model to other countries whose carbon emissions are as much, if not more of, a problem than those of the U.S (e.g. China). If the U.S. it to be a city on a hill, it must act like that and not like a spoiled and entitled child. President Obama had the political courage to acknowledge the seriousness of this issue, even if he was unable or unwilling to press this issue on a national level over the past four years.

You can view that as a promise broken. There’s no reason to view it otherwise. But I view the silence of the Republican party, and its nominee, on this matter as a promise never made. You can’t break a promise you don’t make, but failure to make the promise  at all, when all evidence demands it, is the sign of a party that cannot be trusted to do what is best for an entire nation.

[1] http://www.npr.org/2012/08/30/160357612/transcript-mitt-romneys-acceptance-speech

[2] http://www.npr.org/2012/09/06/160713941/transcript-president-obamas-convention-speech