Well, this is going to suck:
However, I learned something that is either funny, adorable, concerning, or all of the above: Dallas has something called “Ice Force 1,” which is a preparedness state that means they send lots of sanding trucks (did I say “lots”? I meant “30″) out to known trouble spots.
BRACE FOR IMPACT! SET ALERT LEVEL TO “ICE FORCE 1″!!!!
I have a subscription to “The Economist,” but I’ve been so busy lately that I’ve neglected the last few issues. So it was with great interest that I found from an acquaintance of mine that they recently printed an article entitled “How Science Goes Wrong: Scientific Research Has Changed the World. Now it needs to change itself.” I finally had a chance to sit and read the article this morning. Here are my thoughts.
First – what is “science”?
Before a discussion of the article can commence, one needs to first define “science” – failure to do this clearly, as is the case in “The Economist” article – means that arguments made from the undefined term are, at best, guilty of being based on equivocation. “Science” is a process of making observations, proposing testable and falsifiable explanations, testing the explanations, assessing the tests, and disseminating the outcomes for continued testing, verification, and (if useful) application. In essence, “science” is an error-correcting framework for establishing reliable explanations of natural phenomena. It’s an ideal to which anyone seeking a reliable body of knowledge will strive.
It’s also important to explain what science is not. Seeing as today is “Carl Sagan Day,” I’ll begin with an insightful quote from this communicator of science: “Science is more than a body of knowledge; it’s a way of thinking.” Science is not a collection of facts. Science is not something done only by “scientists,” though scientists practice science and strive to achieve its ideal framework and goals. Science is not defined only by what is printed in journals, presented at conferences, or repeated in sexy headlines by the media. It is, rather, a framework that includes those things – facts, observations, publications, presentations – but is bigger than that, ever seeking to correct for mistakes in any of its parts. That may take days, or decades, or centuries.
The Economist Article – a summary
If I were to summarize the article, I would do it thus. They rightly point out that, at the heart of science, we have the concept of “trust, but verify.” The author then argues that science’s success has made it complacent, admitting more bad research than good research. They argue that so much bad science squanders resources. They argue that competition for limited funding has forced scientists to trump up any result in order to grasp a piece of the funding, pushing results that go unverified. They note flaws in review of results. The author suggests ways to improve, such as improving mastery of statistics, publishing protocols ahead of conducting the research itself, and encourage replication of results.
I don’t disagree with the suggestions that the author makes for improving aspects of the practice of science; we already know that each piece is flawed, and only the entire framework, applied consistently and continually, can identify the errors in the parts. Particle physics took the mastery of statistics more seriously in the last few decades, and this has led to tremendous revolutions not only in sifting tenuous from reliable results, but also in the level of discourse in the field about statistics and its intricacies. I find, comparatively, a lot of medical literature laughable in its naivete when it comes to statistical analysis and confounding factors in an experiment. That isn’t to say there is only weak medical science – in fact, the best medical science is marked by its deep appreciation for and application of for statistics (just as the best science in any field is marked by a complete respect for the best data analysis practices). I agree that protocols should be made more public, and I especially agree that the publication and funding system should encourage more replication.
However, The Economist is truly guilty of many fallacies in constructing the argument that science is in trouble. Primarily, they mistake science for its parts. Science is greater than the sum of its parts, because it is a long-term, error-correcting framework that seeks to correct the flaws in its steps. Good ideas survive because they turn out to be useful, not because they are merely published in a journal. It’s nearly impossible to determine usefulness ahead of funding, publication and dissemination, and this is where The Economist truly misses the point: you must ALWAYS admit a high rate of publishing weaker or ineffective ideas, so that the community as a whole can go through the process of assessing and then trying to apply them. If you already know your idea is good, why bother with science? Science is a process to establish reliability, and to determine that you have to see it through.
The author also cherry-picks his way through the entire landscape of science, noting that “prominent” members of a couple of fields decry the level of poor results in journals in their fields. Every scientist in every generation in every field has, at some point, been able to complain about this; the pace of publication is accelerated by our modern toolkit, but that is no excuse to whine about the number of bad results in journals. Bad results have ALWAYS been published in journals or presented at conferences; what matters is that the process of science is still able to sort out, after the fact, the truly useless from the truly useful. Progress results. Progress in science has not stopped; in fact, if anything, the paceo f progress is exponential and positive in many areas. That, more than anything, tells me that science is healthy. It still works. Yes, bad ideas get out there; but what matters is that the few good ideas also get out there, they get used, and they lead to new knowledge. You fundamentally cannot have the good ideas without the bad.
This process has a price. Since it’s impossible to stop every bad idea at the point of publication, one must allow the entire process of science to proceed. Journals can do a better job, but they cannot do a perfect job – if journals were the sole arbiter of truth, one would not need the scientists in the first place. I’d rather that LOTS of ideas get published so that we can catch the few good, useful ones, rather than risking the best ideas solely in the name of reducing the publication rate. Good ideas might be killed prematurely as a result, and that is unacceptable.
Regarding the incentives to cheat to get funding, all systems encourage cheating because no system is perfect. Personally, I admire the academic review and faculty tenure system; it’s brutal and frustrating and grueling and irritating . . . just as it should be. Universities and colleges invest tremendous resources in their faculty, in order to gamble for a few major breakthroughs and a long commitment to excellent teaching. They have every right to expect that this investment is returned, through grants and publications. A physicist, depending on their field, might expect to get a faculty start-up package anywhere between $100k to $5M – yes, that’s right . . . millions of dollars. That’s a serious investment in equipment (labs, machines, supplies, etc.) and people (students, post-docs, staff, etc.). It’s incumbent on the faculty member to utilize those resources to launch a successful research program, and that is stressful. The University expects to have the investment returned when the faculty member, using the start-up as seed money, draws in research grants. It’s ironic that The Economist criticizes this system, since it’s not terribly different from a business investment model that begins with investor seed money and ends with taking the company public. Faculty have to sell their results, that is true; but companies have to sell their product, and in both cases hype is inevitable. What matters is not the hype – what matters is the usefulness of the results; scientists are BRUTAL when they detect all hype and no substance. Sure, that process can takes years, or decades. Find a better system for investing and generating return on investment, with error-correction built in, and we can talk.
The job of the scientist
The job of the scientist is exhausting. You need to advance your own ideas while critically appraising the results of others. It’s a bloodbath. But it’s rewarding, because in 1 year, or 5 years, or 20 years, you’ll have a more reliable body of knowledge than you have now. Maybe you’ll even get to answer one of those pressing questions that drew you into your field in the first place.
That is the goal of science – reliable and useful knowledge. A single result could be crap, or it could be genius – only the meat grinder of the scientific method will sort that out. Scientists are people, and people make mistakes. But science gives us an ideal framework for sorting fact from crap, and we strive to that ideal.
It would be suicide to human knowledge to slow the trickle of bad ideas, because there is no perfect solution for identifying only the good ideas while ignoring the bad ones. Oh wait . . . yes there is such a solution . . . and it’s called “science.” It is practiced by imperfect people, but it’s the best system ever established by our species for generating reliable information. Can it be improved? Sure. Does it need to change? Only if you like your facts to be more useless.
Author’s Notes: I’ve updated the original post to list the news agencies that reported on this as if their audiences should accept it as fact. I only selected from news agencies with a national reach or an ostensibly scientific mission – those that have the resources to know better and be more critical in reporting “emerging medical research.” I also add a list of news agencies that got it right – they critically and skeptically appraised the claim in the larger context of markers of addition, study design, etc.
I have also edited my comment on blinding to indicate that it’s unclear whether they used it, and the fact that it’s not mentioned is a red flag.
I also added a comment on misuse of reasoning in drawing the conclusion about cocaine addition.
I know that some people object to my title – however, it’s intentionally provocative to indicate that failing to consider rival causes can lead you down a potentially wrong path when drawing conclusions from data. Seriously – what if the rats just found the rice cakes disgusting and derived an inevitable and strong reward from a food source that wasn’t disgusting?
I saw this headline all over the place today – here is one representative example:
Oreos May Be As Addictive As Cocaine; That stuf is addictive 
Here is another one:
Addicted to Oreos? You truly might be. 
Wow! Is that really what this study found? Nope. The only thing these researchers proved is that given a choice between Oreo cookies and rice cakes, mice choose Oreo cookies. Is that really a surprise? Let’s take a closer look.
The study was conducted by a group of Connecticut College students supervised by a professor of psychology . The two students are undergrads – they are not trained or credentialed scientists, but are certainly in training to be scientists. This is excellent – that students are learning to construct experiments, gather data, and analyze results – but one should be cautious in just accepting the conclusions of their wok. Their work has not been peer reviewed, nor has it yet been presented at a conference and subjected to feedback from the community.
They trained the rats to run a maze. On one side of the maze they offered the rats rice cakes, and on the other they offered them Oreos. They compared the amount of time the rats preferred to spend on the different sides of the maze. They then compared that data to data from a separate trial of rats comparing how much they prefer a saline injection to a cocaine injection. You can already see the design flaws here. Why rice cakes? Why not something rats normally like to eat? Or, for that matter, if you’re going to compare to cocaine . . . why not cocaine? You can begin to pick apart the premise of this experiment in a heartbeat.
So the data gathering is fatally flawed. What makes it worse is the conclusion drawn by the faculty member supervising this work:
“Our research supports the theory that high-fat/ high-sugar foods stimulate the brain in the same way that drugs do,” Schroeder said. “It may explain why some people can’t resist these foods despite the fact that they know they are bad for them.”
Professor Schroeder fails to consider rival causes – the research also supports that Oreos are more attractive than rice cakes to the rat palate. So . . . how did they draw the cocaine-related conclusion? The students compared a marker indicating neural activity in response to stimuli and checked it against the presence of that marker in a cocaine-addicted rat brain.
[The students] used immunohistochemistry to measure the expression of a protein called c-Fos, a marker of neuronal activation, in the nucleus accumbens, or the brain’s “pleasure center.” – “It basically tells us how many cells were turned on in a specific region of the brain in response to the drugs or Oreos,” said Schroeder. They found that the Oreos activated significantly more neurons than cocaine or morphine.
What’s unclear is: were the studies of the chemical markers done blind (e.g. did the students KNOW that they were analyzing brain chemistry from Oreo-conditioned rats)? Did the students bother to see if ANY conditioning, using Oreos, or cocaine, or any other pair of choices, led to the same brain chemistry? This whole thing seems like a really bad oranges-to-apples comparison
Is this reliable science?
Nope. The science reporting on this has been terrible, to boot. I applaud students getting into research, but I find deplorable the fact that their school made a press release about unreviewed research and cited it like gospel.
This, as presented, is not reliable science because:
- It’s unclear that the study actually compared the things that it then claimed to compare in the conclusion – Oreos and cocaine. It merely trained rats to prefer one of two choices and then looked at their brain chemistry for markers of addiction. That’s nice, but it’s not at all supportive of their conclusions.
- Regarding the conclusion, the reasoning applied here is based on incorrect logic. One can frame the independent observation – that addiction leads to the stimulating of reward centers in the brain – as a syllogism: “If a rat is addicted, then the pleasure center of its brain will be stimulated.” One can then take the conclusion from the above study and see if it affirms the consequent or affirms the antecedent. The conclusion that the professor in charge of the study draws is, “Based on the data, the rat brain pleasure center was stimulated.” That’s a correct conclusion. However, the next step in the reasoning is wrong. The professor then concludes, “Since the rat brain pleasure center was stimulated, the rat was addicted to Oreos.” This is a wrong form of logical reasoning called “Affirming the Consequent.” For instance, consider the following syllogism: “If it is a car, it has wheels.” Now, affirm the consequent: “I observe something with wheels.” One cannot then conclude, “This thing I observe is a car.” That is wrong. It might be a skateboard, or a cart, or something else with wheels. Just because it has wheels doesn’t make it a car. Just because the rat brain pleasure center is stimulated, doesn’t mean the only cause is addiction.
- Addiction is more than just preferring A over B – it’s also withdrawal symptoms and a host of other associated risks and problems. Did the students then withdraw Oreos and observe whether or not they executed the same withdrawal behaviors as cocaine-addicted rats?
- There was no reported attempt to blind the study – that raises a red flag. At the very least, the authors should have indicated key features of their study in the press release, since there is no scientific supporting documentation. Students analyzing brain chemistry should NOT know what rat brains – Oreo-conditioned or non-conditioned – they are measuring. They should have been asked to see if there were any populations present in their data. If they identified two, they should have then determined if the populations correlated with the Oreo-conditioned rats and normal rats; or, if the Oreo-conditioned rats are then comparable to cocaine-addicted rats. It will be interesting to see whether or not they conducted the study blind, to remove bias.
- The statistics are unclear. How many rats? What are the errors on their chemical measurements? Did they even bother to assess uncertainty?
- It’s unreviewed. It’s unverified.
At best, this study concludes that rats, like humans, prefer Oreos over rice cakes. Surprise. Rice cakes taste like shit.
News Sources That Should Have Been More Critical and Skeptical But Were Not:
Christian Science Monitor: http://www.csmonitor.com/Science/2013/1016/Oreos-addictive-Rats-treat-Oreos-like-cocaine-study-suggests
Discovery News: http://news.discovery.com/human/health/oreo-cookies-as-addictive-as-cocaine-131016.htm
ABC News: http://abcnews.go.com/Health/oreos-addictive-cocaine/story?id=20590182
Hartford Courant: http://articles.courant.com/2013-10-15/news/hc-oreo-addictive-rats-1016-20131015_1_rice-cakes-one-experiment-humans
LA Times: http://www.latimes.com/food/dailydish/la-dd-oreo-cookies-addictive-cocaine-20131016,0,3166408.story
CBS News: http://www.cbsnews.com/8301-204_162-57607785/oreos-may-be-as-addictive-as-cocaine-morphine/
Chicago Tribune: http://www.chicagotribune.com/business/technology/chi-nsc-why-your-brain-treats-oreos-like-a-drug-20131016,0,1494058.story (regurgitated the Forbes story – for shame)
News Venues that Were More Critical and Skeptical
The Daily Mail claims in their science section that the 60% increase in arctic ice extent comparing August of 2012 to August of 2013 means “global cooling” is happening. But is this bad science reporting? Yes. This claim cherry-picks data, comparing only August of 2012 to August of 2013. The article ignores absolute numbers over relative (percentage based) ones, and ignores uncertainty year-to-year in expected average ice coverage, as well as the long-term trends in ice coverage. This article is riddled with the worst kind of pseudoscience.
What is climate, and how is that different from weather?
Climate is the effect of weather aggregated over decades. Weather is the variation in environmental temperature and moisture patterns in a week, a month, a year, and even several years. A lot of news agencies, TV talking heads, and other preachers at their bully pulpit confuse these issues and mistake changes in weather patterns in one or two years for climate.
This mistake has led many in the press to declare all kinds of wrong things about the weather. Hemispherical snobbery has also led people to make mistakes like the declaration that the 1960s-1970s were a time of “global cooling.” Thanks in large part to the dispersion of aerosols into the Northern Hemisphere through the prior decades, “global cooling” was actually confined largely to the Northern Hemisphere that wrought those aerosols and particulates; the Clean Air Act in the U.S. removed those particulates, and the Northern Hemisphere resumed a warming trend that had been mitigated by pollution in the north but was largely unchecked in the Southern Hemisphere (c.f. my old post on this ).
The Daily Mail has proclaimed, in what is ostensibly their “science” section, that “And now it’s global COOLING! Record return of Arctic ice cap as it grows by 60% in a year.” That might seem impressive – after all, the Arctic Ice Sheet is huge. But the Earth is huger. This is a good example of mistaking something that’s big by human standards as being also big by Earth standards. Such a mistake in framing the scale of things is the first flaw in this piece.
The major flaw, however, is a total misunderstanding of (1) the absolute, not relative, numbers involved (percentages sound impressive, but when it comes to something like sea ice it’s absolutes that matter), (2) trends in climate (which manifest over decades, not in a year), and (3) uncertainty on single-year measurements. It may seem impressively unlikely that the ice sheet could grow by so much in one year; after all, humans live about 80 years (thanks entirely to evidence-based medicine), and such big growth in one year might seem impressive to such a short-lived creature as a human. But the Earth is 4.5 billion years old, and 1 year is as impressive to the Earth as half-a-second is impressive to a human being. That’s the equivalence of one year in the timescale of the Earth to a human being – 0.5 seconds. Would you, in what happens in just one 0.5s of your life, make pronouncements?
Keep in mind that the Daily Mail is comparing numbers using monthly values, picking two months separated by one year. This is akin to making a judgement based on events that occur in less than 0.05s of your life!
The trend in Arctic Ice – monthly, yearly, and by the decade
NASA keeps excellent data on the trends in Arctic Sea Ice extent , specifically the area of the ice in units of “millions of square- kilometers,” which I’ll denote as “mk2.” From 1979 to 1984, a five-year period, the average coverage in sea ice was about 6.6 mk2. Year-to-year variations during that period were at the level of up to 0.5 mk2 each year.
Sea ice extent (area) varies month-by-month. It can be big during the Arctic winter months (e.g. 15 mk2) and small during the Arctic summer months (e.g. 5 mk2), so in just 6 months the area can swing by huge amounts. It’s better, as a result, to compare yearly and even multi-yearly means of the ice area, rather than month-by-month numbers. The variations are too large, on a month scale to make ANY conclusions.
Looking at the data, we see the trend takes decades to reveal, but it’s clearly downward and the average amount of sea ice extent is decreasing at an accelerated rate. While the year-to-year average area variations from 1979-1984 were at the level of 0.5 mk2, the total yearly mean sea ice area has dropped from an value of about 6.6 mk2 during that period to about 3.6 mk2 over the last 5 years, excluding 2013 (which is not done yet).
So the drop in mean sea ice coverage has been about 3 mk2 in just about 30 years, or about 1mk2 every decade (on average . . . keep in mind the extent has decreased at an accelerating rate, so most of that loss has actually only been in the last 15 years).
The Daily Mail chooses to quote their claim using relative (percentage) numbers, cherry-picking two specific months rather than using absolute numbers and yearly averages. They claim that sea ice coverage increased 60% between August of 2012 and August of 2013. However, a look at the raw data for 2012 and 2013  shows that for most of 2012 and 2013, the areas have been about the same! It’s only since May that the two sets of numbers diverge, but remember – monthly variations are LARGE and it’s difficult to make conclusions month-by-month using such noisy numbers. Better to take the arithmetic mean, and look at the trend.
So I did that. The trend is clear. The average sea-ice extent in 2012 from Jan-Aug was 11.9 mk2, while the same average in 2013 was 12.1 mk2. That’s not only within the month-by-month noise; it’s within the yearly average noise, too.
This is nicely illustrated by a graph of the averages and the monthly and yearly trends from the National Snow and Ice Data Center :
What about the percent increase? 60% sounds like a big number . . . except that it’s 60% of the smallest ice extent number on record. In fact, it’s not even clear where the Daily Mail’s 60% comes from . . . because it doesn’t come from the data itself. The NSIC numbers report 4.71 mk2 in August 2012 and 6.09 mk2 in 2013. That’s a 29% increase, comparing only those two months. Where they got 60% is unclear.
But, that said, it’s still not impressive. The uncertainty on any single monthly ice extent value, averaged over 1981-2010, is about 1 mk2. That means that monthly variations at the level of 2 mk2 are well within statistical noise (95% confidence level, assuming Gaussian errors). This is a variation of 29% of the 2012 value in August . . . or about 1.3 mk2. That’s pretty unimpressive, as it’s well within the noise envelope.
The Daily Mail article draws wild conclusions about climate by mistaking weather for climate, cherry-picking two monthly arctic ice area values, quoting relative and not absolute numbers, and ignoring uncertainty in measurement. Accounting for those issues, one cannot conclude that “global cooling” is happening based on a single year, and certainly not a single month in a single year. Such a claim ignores the decadal trends in ice extent and basic critical thinking skills. With such blather, The Daily Mail is promoting dangerous pseudoscience and weak sense critical thinking.