Remembering the hits, forgetting the misses

Tim Tebow Demotivational Poster
OK, call me snarky. But the numbers don't lie: there is nothing remarkable about Tebow. This is a good lesson in remembering the hits and forgetting the misses. Graphic from Ref. 6.

[Author’s Note: After writing and posting this, the Broncos went on to face the New England Patriots in their final playoff game (the 18th of the Broncos’ season). The Broncos lost SPECTACULARLY, 10-45. This makes their record of wins/losses now to be 9/9, exactly 50% . . . the coin flip would have been just as good in determining the outcome of the Broncos’ season as the actual season. ]

Let me say that I have nothing against Tim Tebow. He seems like a fine young man. He’s certainly very religious; although, ironically, he behaves in a way strongly inconsistent with the teachings of the very person to whom he proclaims a public faith: Jesus. Recall that, according to the Bible, Jesus taught:

And when thou pray, thou shall not be as the hypocrites are: for they love to pray standing in the synagogues and in the corners of the streets, that they may be seen of men. Verily I say unto you, They have their reward. But thou, when thou pray, enter into thy closet, and when thou hast shut thy door, pray to thy Father which is in secret . . . [Matthew 6:6 & 7]

The public reaction to Tim Tebow, however, is the pseudoscience gift that keeps on giving. As I discussed in my last post about him [1], he was hailed in the sports press with mixed reviews (both as a terrible quarterback and as an exciting quarterback). He  had captured media stardom with a combination of excessive public religiosity and “sudden comebacks” (more on that below). But overall his performance was statistically uninteresting. As I noted, the Broncos’ rate of winning 1, 2, 3, or 4 games in a row as “comebacks” followed a roughly exponentially decreasing trend (the rate of winning 2 in a row by comeback is half that of 1 in a row; the rate of 3 in a row is about half that of 2 in a row; etc.). Based on that trend, the Broncos were right on track for 4 comebacks in a row, whether or not Tebow was the QB.

Now that the Broncos have completed a lot more of the current season, we can really look at Tebow and see how he ranks up against the “greatest” quarterbacks in NFL history. Let’s see what we find.

First, we need some “great” quarterbacks. I want ones from the last 30 years, so that they are playing in the most modern context of American Football. Here is the opinion of Jason Whitlock at Fox Sports [2]:

  1. John Elway (Denver Broncos)
  2. Joe Montana (San Francisco 49’ers)
  3. Johnny Unitas (Baltimore Colts)
  4. Dan Marino (Miami Dolphins)

Based on my own knowledge of Football history, I wouldn’t disagree with these. Conveniently, Tebow and Elway play for the same team. I will ignore Unitas because of how long ago he played the game.

I am primarily interested in their QB stats in the first 2 years of play, to see whether Tebow is really an anomaly compared to the greats. What are the good stats? One men’s magazine article claims the most important QB statistics are:

The single most important statistic for judging quarterbacks is yards-per-attempt (YPA). This illustrates a player’s efficiency. Even though they are highly correlated, it is also important to look at completion percentage as well. This is more indicative of consistency and is not as related to the ability of receivers to run after the catch. Along with YPA and completion percentage, quarterbacks are successful if they do not turn the ball over. Again, everything is relative, so we must account for the total number of pass attempts. Interceptions per 500 pass attempts is a stat that is easy to compare. After those items, mobility is very valuable. [3]

Here are some stats [4].

Yards Per Attempt (YPA) in first two years

The first number is their first year of play, the second number is the second year of play. The errors are obtained by propagating the Poisson uncertainties on the Yards Gained by Passing and the Number of Attempted Passes.

  1. Elway: (6.4 +/- 0.4) and (6.8 +/- 0.4)
  2. Montana: (4.2 +/- 1.0) and (6.6 +/- 0.4)
  3. Marino: (7.5 +/- 0.5) and (9.0 +/- 0.4)
  4. Tebow: (8.0 +/- 0.9) and (6.4 +/- 0.4)

The numbers are very revealing. The error-weighted average of the “best QB” YPAs is (7.17 +/- 0.03), while Tebow has a YPA of (8.0 +/- 0.9) in year 1 and (6.4 +/- 0.4) in year 2. Unlike the great QBs, Tebow’s performance decreases in his second year (albeit with a large uncertainty on the YPA in his first year due to the few passes he got to throw in his first season). In Tebow’s second season, his is well below the average of the greats.

Interception Percentage (Interceptions per attempt to pass) in first two years

The first number is their first year of play, the second number is the second year of play. The errors are binomial.

  1. Elway: (5.4 +/- 3.6)% and (3.9 +/- 3.8)%
  2. Montana: (0.0 +/- 0.0)% and (3.3 +/- 3.0)%
  3. Marino: (2.0 +/- 2.4)% and (3.0 +/- 4.1)%
  4. Tebow: (3.7 +/- 1.7)% and (2.2 +/- 2.4)%

In general, great QBs have interception rates consistent with zero within the binomial errors. Tebow exhibits the most significant interception rate (in his first year). Compared to other great QBs, his rate is higher than 2/3 of the other three and in his second year is lower than all three. This is one place where Tebow superficially looks good, but when you take into account uncertainties it’s tough to really distinguish anything here.

Pass Completion Rate (Percentage)  in first two years 

The uncertainties are binomial. The first number is their first year of play, the second number is their second year of play.

  1. Elway: (47.5 +/- 6.5)% and (56.3 +/- 4.5)%
  2. Montana: (56.5 +/- 18.3)% and (64.5 +/- 4.5)%
  3. Marino: (58.5 +/- 4.9)% and (64.2 +/- 3.1)%
  4. Tebow: (50.0% +/- 11.0%) and (46.5 +/- 6.5)%

The great quarterbacks tended to complete passes at a rate better then 50/50; that is, better than flipping a coin and deciding whether the pass is completed based on complete chance. Elway is the exception, although he improved considerably in his second year. Two of the top 4 greatest quarterbacks achieved completion rates well above 50% by their second year (about 64% for Montana and Marino).  In all cases, the great quarterbacks IMPROVED their pass completion rate by their second year.

Tebow bucks the “trend of the greats” in a couple of ways. His performance WORSENS in his second year (albeit consistent with a flat performance within uncertainties) and it’s never better than flipping a coin. In fact, by his second year you could have flipped a coin every time he made a pass and have completed more passes by calling “HEADS!” than by letting Tebow’s pass actually decide whether completion occurs.

Summary

Is Tim Tebow a remarkable QB? Not really, according to empirical data. In two of the three major statistical categories, Tebow is not at or above the greatest recent quarterbacks; in the third, his ranking is indeterminate. His pass completion rate is no better than a coin flip, and in fact was WORSE this year. His interception percentage may be a bit smaller than others, but the uncertainties are large on all numbers presented. His Yards per pass attempt are unremarkable compared to the great quarterbacks. In two of the three stats categories, his performance worsened in his second year whereas the great quarterbacks improved.

Looking at the Denver Broncos total record, they played 17 games. Of those, they won 9 and lost 8. Their rate of winning games was slightly better than the rate of getting heads or tails in a coin flip. Certainly, considering just 17 such coin flips their performance is completely consistent the rates of heads and tails over many successive groups of 17 trials. They had one “streak” of 4 comebacks mid-season, which was then followed by 4 successive losses (which nobody seemed to call a “loss streak” initiated by Jesus’ displeasure with Tim Tebow). In fact, everything about the Broncos’ performance in 2011 is consistent with a sequence of coin flips, including “apparent streaks” of both wins and losses. This is all consistent with random numbers.

So I don’t see what all the fuss is about. Tim Tebow’s throwing arm is about as interesting as flipping a coin. If the fuss is about his religiosity, then the numbers suggest prayer isn’t helping him. But then, controlled studies of prayer also tell us that this practice confers no measurable benefits [5]. So, pretty much, this guy is uninteresting in both a secular and a religious sense.

I suspect that Tebow’s popularity is due more to the problems that we, as humans, have with data. We remember the hits (the Denver Broncos won 4 games in a row, by comeback, with Tebow as QB) and forget the misses (the Denver Broncos lost 4 games in a row, with Tebow as QB). That is why it is always important to evaluate a situation not based on personal biases (e.g. liking Tebow because he prays in the end zone) but on empirical evidence. The evidence says he’s not a great QB. Maybe, with more practice, Tebow will improve. I hope so, since he seems like a nice guy. But a nice guy is not necessarily a great QB.

[1] http://steve.cooleysekula.net/blog/2011/12/17/the-tebow-distribution/

[2] http://www.askmen.com/top_10/sports_100/135_fitness_list.html

[3] http://msn.foxsports.com/nfl/story/the-10-greatest-qbs-of-all-time

[4] http://www.pro-football-reference.com

[5] http://articles.latimes.com/2006/mar/31/science/sci-prayer31

[6] http://www.buzzfeed.com/treyc/tim-tebow-crying-motivational-poster-uw5

 

Anecdon’t

HPV Virus
A group of koilocytes on the the bottom right is accompanied by two normal intermediate squamous cells at the top and left from Ref. 19.

The Republican Presidential Candidate debates (and all the media that encircles them) are a great place to look for examples of poorly applied thinking. Specifically, it’s a buffet of examples of pseudo-scientific argument. In this post, we’ll use one example that stemmed from the most recent CNN/Tea Party Express debate to highlight the danger of the anecdote.

First, what is an “anecdote?” The Merriam-Webster dictionary provides the following definition: “a usually short narrative of an interesting, amusing, or biographical incident.” [1] According to that same source, the word derives from the Greek word anekdota, which means “unpublished items.”

Anecdotes are a popular staple of the campaign trail. Candidates of all stripes use them to make themselves appear more relatable to their audience. They seem to be used to serve the role of a kind of parable – an anecdote usually is meant to contain a meaning or truth upon which the candidate builds an argument or with which a candidate counters their opponent.

Anecdotes are not a form of scientific evidence. Why not? First, they are usually second-hand information, at best. The person telling the story is almost never the participant in the story; rather, the story is about somebody else. You’re lucky if the anecdote resulted from a direct eye-witness event.

Second, an anecdote is a form of witness testimony that may be first-hand, but more likely has been transferred by an oral tradition from person to person (with the usual change in details or embellishment of events resulting from the transfer). Witness testimony is a form of legal evidence but is not a form of scientific evidence. Why? People lie. Even when they do not lie, people mis-remember events, or miss the full detail of an event, or both. Even when they do accurately recount an observation, they may assign causation in the sequence of events even though the relationship between the sequence may be accidental, coincidental, or completely random. The point is that people are fallible; the eye is fallible, the ear is fallible, and the mind is fallible. A single story cannot then be relied upon as the sole basis of a scientific argument. It may be the seed of an evidence-gathering cycle, but in and of itself the anecdote – a form of witness testimony – is not primary evidence. For an overview of the the fallibility of eyewitness testimony, see Ref. [2].

With all of this in mind, let’s look at the fallout from an exchange between Presidential Candidates Rep. Michele Bachmann and Gov. Rick Perry [3]. The exchange focused on Rick Perry’s executive order requiring young women (albeit allowing for parental opt-out of the requirement) to be vaccinated against the Human Papillomavirus (HPV). HPV is strongly linked to cervical cancer, is carried by men and women, and is sexually transmitted. That executive order was overturned by a veto-proof majority of the Texas Legislature. NPR ran an excellent story on this strange episode: a staunchly Republican Governor, professing support for individual liberty, issued an order requiring young women to receive a vaccine (albeit with a parental opt-out) [4].

This post is NOT about the political, policy or rights issues raised by his decision (however, see the “footnote” near the end of this post). Rather, this post is about something Michele Bachmann said on the “Today Show” on the morning after the debate [5]. She was discussing her position on the vaccination of young women against HPV.  She relayed an anecdote told to her by a debate audience member after the event.

” . . . all innocent little 12-year-old girls or 11-year-old girls in the state of Texas would be forced  by the government to take an injection of what could potentially be a very dangerous drug . . . I had a mother last night come up to me last night here in Tampa, Florida after the debate. She told me that her little daughter took that . . . vaccine and suffered from mental retardation thereafter. It can have very dangerous side-effects . . .  people have to draw their own conclusions.” (Ref. [5])

Let’s parse that argument. Bachmann states her thesis – that the vaccine could be potentially dangerous. She then cites her evidence: a single anecdote from a single mother that connects two events, the receiving of the vaccine and the onset of some kind of mental retardation. Bachmann then repeats her thesis, and leaves the final decision to an audience that, one assumes from her argument, she believes she has now informed with this argument. This unit of speech represents the fullness of her argument.

What is missing here? She cites no medical literature to back up her statement. Let’s look at the literature on this issue.

There are some relevant numbers in making an informed assessment of the risks associated with vaccination. First, what is the risk from the disease prevented by the vaccine? The numbers I cite here are all taken from Ref. [6], an article entitled “Epidemiologic Classification of Human Papillomavirus Types Associated with Cervical Cancer” from the New England Journal of Medicine. This article has been cited over 2600 times. The authors pooled data from 11 case-control studies of cervical cancer, including women with and without symptoms of the cancer.  Almost 91% of the women in the group with cervical cancer also had HPV in their bodies (evidenced by the detection of HPV DNA in cervical cell samples), whereas only about 13% of the control group had positive results for HPC DNA in their cervical cells). Cervical cancer was once the leading cause of female cancer deaths in the U.S.; in the last 40 years, the rate of death from this cancer dropped significantly due to screening (e.g. Pap tests). In 2007, the most recent year for which data are available [7], over 12,000 women were diagnosed with cervical cancer. Since the cancer is linked strongly to a virus, and the vaccine prevents against the virus, about 90% of these cases were preventable.

Based on 2007 census numbers, this number of cases corresponds to 0.008% of the female U.S. population (and thus 0.004% of the U.S. population). Each year, about 0.001% of the U.S. population dies from cervical cancer.

Let’s pause for a moment and compare this vaccine and its associated preventable disease with some other vaccines and their preventable diseases.

  1. Polio [8]: During the period of highest infection rates in the U.S., an average of 35,000 cases of polio were reported per year. In 1954, the last year before the vaccine was introduced, this represents 0.02% of the U.S. population [9]. Of that infected population, 72% have no symptom  but spread the virus; 24% exhibit minor infection-related symptoms, such as fever, upset stomach, or flu-like symptoms, but do not develop paralysis; only 1% of infected people develop the iconic paralysis that put fear into the nation. If polio-induced paralysis is the symptom that was most feared, and thus the thing most worth preventing when requiring vaccination, then we are talking about 1% of 0.08% of the population, or 0.0008% of the population that was most seriously affected. Was polio worth preventing with a potentially risky vaccine just to save 0.0008% of the population from paralysis? The answer is an unequivocal “yes.”
  2. Pertussis (whooping cough) [10]: From 1940-1945, there were about 1 million reported cases of whooping cough. This is an average of 200,000 cases per year. This is 0.14% of the population in 1945. Of those infected by whooping cough, the mortality rate in the 1940s was about 0.0025% [11].  The vaccine was introduced to prevent the disease in a significant fraction of the population (certainly a very significant fraction of infants, as most risk for mortality from the disease), and to save the lives of just 0.0000035% of the population who would otherwise die from the disease.

These are just a few examples. We see that due to the nature of the death, or the severity of the disease, or the communicable nature of a disease, we have acted as a people to develop and distribute vaccination for diseases that kill far fewer people than are killed by cervical cancer.

But it is not enough to discuss the incidence or mortality rates of a disease; we should also consider the complications associated with giving the vaccine to the population. For instance, what are the risks associated with giving every woman in the U.S., 12 years or older, the currently available vaccines for HPV?  As part of the evaluation of the drug Gardasil, data were collected using a randomized double-blind study of patients [14]. The findings were obtains from 277 women receiving the actual drug and 275 receiving placebo preparations; the groups were randomly determined. No life-threatening or quality-of-life altering side-effects were reported in either group. Standard injection related effects were reported in both groups. For instance, 83% of patients receiving the actual drug reported ” erythema, pain, and swelling, with severe intensity” [14] while 73.4% of placebo recipients reported the same effects. The margin of error on these numbers is +/- 2.7% and +/- 3.6%, respectively (I computed my own binomial errors on these numbers using the numbers presented in the references). These two numbers agree within 2.1 standard deviations, which is not a significant difference between the two groups. Most vaccines yield some kind of injection site effect, and here we see that the placebo causes just as many as the drug.

The CDC information site on the vaccine notes that the medical community continues to monitor the women who receive the vaccine for evidence of very rare complications that won’t show up in such trials. But the American Academy of Pediatrics issued a statement following Bachmann’s interview on the “Today Show” stating that her comments were irresponsible and fly in the face of any actual medical and scientific evidence about complications associated with the vaccine [15]. They state that 35 million doses have been administered with no scientifically demonstrated causal link to serious side effects.

Let’s take stock of what we have learned. Primary evidence gathered in randomized, double-blind trials shows that one of the two approved vaccines for HPV, Gardasil, is not only efficacious (90% success rate in preventing infection by HPV) but safe (no significant difference between placebo and the vaccine in studies of side effects). There is no additional data that clearly links rarer complications to the specific vaccine. The risk associated with taking the vaccine  is small (35 million vaccinations given with no evidence of serious side effects, suggesting a life-altering complication probability of  <0.000003%) compared to the fact that death from cervical cancer results in 0.001% of the population.

Bachmann offered only an anecdote as evidence of the need to proceed cautiously with policy regarding HPV vaccination. Anecdotes are a form of eye-witness testimony, and thus unacceptable as primary scientific evidence due to well-documented problems with this form of evidence. Since then, she has reportedly been offered thousands of dollars to produce primary scientific evidence backing up her claim (in the form of wagers from medical researchers) [16]. So far, her campaign has not responded.

We have an excellent teachable moment here. If you want to win the war with facts, you need to provide them. Testimony are insufficient as facts. Anecdotes, then, are anecdon’ts when it comes to making a solid argument.

Footnote

Let’s step away from the science issue for a second and think about the policy issue. Whether or not to require that 11-12 year-old girls received the HPV vaccine is a policy question. There is a science component to this issue. We need to set aside the values issues for a second (e.g. different people, with different ethical or moral world views, will respond differently to the question of whether or not a young woman should receive a vaccine for a sexually transmitted disease, independent of what the science says).

The recommendation about the age of vaccination stems from a couple of pieces of data: how many vaccinations are needed to receive complete protection and when a young woman typically becomes sexually active. The CDC has statistics on sexual activity [17]. For instance, by age 15 about 8% of women engage in oral intercourse and 26% of women engage in vaginal intercourse; rates are similar for young men. At age 14, 5.7% of young women claim to have had sex. According to the Kaiser Family Foundation [18], the median age at which first experience with sexual intercourse occurs is about age 16; that means 50% of young girls have their first sexual intercourse BEFORE age 16. If prevention is the goal, the vaccine has to be administered at an age before which the majority of sexual activity occurs.

Perry’s executive order followed the recommendations of the CDC: begin the vaccination process at age 12 because it takes about 1 year to receive all three recommended vaccinations. That means full protection from HPV occurs at around age 13. That assures that young girls are protected against HPV by the time they become sexually active.

That’s the science that informs the policy discussion.

[1] http://www.merriam-webster.com/dictionary/anecdote

[2] The Problem With Eyewitness Testimony from Stanford Journal of Legal Studies

[3] http://archives.cnn.com/TRANSCRIPTS/1109/12/se.06.html

[4] “In Texas, Perry’s Vaccine Mandate Provoked Anger,” National Public Radio, “All Things Considered,” http://www.npr.org/2011/09/16/140530716/in-texas-perrys-vaccine-mandate-provoked-anger

[5] “The Today Show,” September 13, 2011. http://today.msnbc.msn.com/id/41521818/#44499076

[6] “Epidemiologic Classification of Human Papillomavirus Types Associated with Cervical Cancer”, N Engl J Med 2003; 348:518-527February 6, 2003. http://www.nejm.org/doi/full/10.1056/NEJMoa021641

[7] CDC, “Cervical Cancer Statistics.” http://www.cdc.gov/cancer/cervical/statistics/

[8] CDC, “Polio Disease – Questions and Answers.” http://www.cdc.gov/vaccines/vpd-vac/polio/dis-faqs.htm

[9] http://www.census.gov/popest/archives/1990s/popclockest.txt

[10] http://www.cdc.gov/vaccines/vpd-vac/pertussis/default.htm

[11] http://www.healthsentinel.com/joomla/images/stories/graphs/us-deaths-1900-1940.jpg

[12] CDC. “HPV Vaccine Safety.” http://www.cdc.gov/hpv/vaccinesafety.html

[13] Villa LL, Costa RL, Petta CA, et al. Prophylactic quadrivalent human papillomavirus (types 6, 11, 16, and 18) L1 virus-like particle vaccine in young women: a randomised double-blind placebo-controlled multicentre phase II efficacy trial. Lancet Oncol 2005; 6:271–278. http://www.ncbi.nlm.nih.gov/pubmed/15863374

[14] http://onlinelibrary.wiley.com/doi/10.3322/canjclin.57.1.7/full

[15] http://www.aap.org/advocacy/releases/hpv2011.pdf

[16] http://www.cbsnews.com/8301-504763_162-20107489-10391704.html

[17] http://www.cdc.gov/nchs/data/ad/ad362.pdf and “Teenagers in the United States: Sexual Activity, Contraceptive Use, and Childbearing, 2002” (PDF). Vital and Health Statistics. National Center for Health Statistics. 2002. Retrieved 2008-04-29.

[18] http://www.kff.org/youthhivstds/upload/U-S-Teen-Sexual-Activity-Fact-Sheet.pdf

[19] http://www.flickr.com/photos/euthman/2825069064/