Bad data analysis: motorcycle helmet effectiveness

My SO came across a Florida newspaper article (Yahoo! News version) that looks at the effect of Jeb Bush's repeal (in 2000) of Florida's motorcycle helmet law. The article (Neale 2006) is based on an analysis of federal traffic statistics; included below is the article's primary data (and graph):

A FLORIDA TODAY analysis of federal crash statistics for Florida revealed a drastic upward spike in motorcycle fatalities involving riders without helmets since the repeal took effect. Annual statewide "unhelmeted" fatalities mushroomed from 22 deaths in both 1998 and 1999 to 250 deaths in 2004, the most recent data available.

That represents an 11-fold increase. By comparison, 270 Florida riders without helmets were killed during the entire 1990s, when the practice was illegal.

But by the same token, motorcycle registrations shot upward 87 percent since the helmet repeal took effect. Annual registrations increased from 238,229 to 445,896 from 2000-04, the Florida Department of Highway Safety and Motor Vehicles reported.

Total Florida motorcycle fatalities have increased statewide since the helmet law's repeal. The yearly death toll leaped from 259 in 2000 to 432 in 2004 -- a 67 percent jump -- National Highway Traffic Safety Administration data shows.

Let's take a look at this in a bit more detail. The newspaper is attempting to test two contrasting hypotheses:

1. Riders who wear motorcycle helmets are less likely to die in accidents than riders who do not wear helmets (and thus Bush's repeal of the 2000 law has led to more motorcyclist deaths).
2. Riders who wear helmets are equally or more likely to die in accidents than riders who do not wear helmets (and thus factors other than Bush's repeal of the 2000 law explain the increase in motorcycle mortality).
   

The number of deaths of non-helmeted riders has risen since the repeal of the law in 2000, and thus hypothesis 1 seems to be supported. However, there are multiple confounding factors that the newspaper has not taken into account, the two largest of which are:

• The frequency of riders who wore helmets may have changed across the study period
  
• The number of miles driven by motorcyclists may have changed across the study period

Since the motorcycle helmet law was repealed in 2000, it's logical to expect that a much larger fraction of riders are now riding without helmets; this could explain the change in proportion of riders dying with and without helmets. And, if the number of motorcycle miles being driven increased over the study period (as the number of registrations did), there would likely be a proportional increase in the number of fatalities, which could explain the increase in total mortality over the study period.

The article doesn't include any data that adequately address these confounding factors. The only attempt at addressing these comes in the number of motorcycle registrations, but registrations may or may not be related to miles driven. Without information on these confounding factors, the data presented are functionally meaningless; we can't use them to differentiate between the two hypotheses.

The National Highway Traffic Safety Administration (NHTSA) has published many reports on the effectiveness of helmets; these reports include datasets that circumvent (or control for) common confounding factors. In their most recent analysis (Deutermann 2004), the NHTSA collated data on fatalities in motorcycle accidents between 1993 and 2002 that involved two people riding on the same motorcycle. When two people ride on the same motorcycle, there are four possible helmet-wearing combinations:

• Neither the rider (person in control of the motorcycle) nor the passenger is wearing a helmet
• The rider is not wearing a helmet, but the passenger is
• The rider is wearing a helmet, but the passenger is not
• Both the rider and the passenger are wearing helmets

Since both individuals are involved in the same accident, by comparing the ratio of rider to passenger deaths across the four conditions it's possible to test the effectiveness of motorcycle helmets in preventing death. This is exactly what Deutermann (2004) did; these results can be seen in Table 1.


Table 1: Data from a 2004 Department of Transportation analysis (Deutermann 2004) of fatalities when there was a pair of riders on a single motorcycle in a single accident. Data are organized by which (if any) of the occupants of the motorcycle were wearing helmets at the time of the accident.

Year	Helmet Used		Number of Deaths		Fatality Ratio
	Rider	Passenger	Rider	Passenger	Rider/Psgr	Psgr/Rider
Total (1993-2002)	No	No	1,107	1,027	1.078	0.928
	No	Yes	141	59	2.390	0.418
	Yes	No	71	85	0.835	1.197
	Yes	Yes	991	869	1.140	0.877

While riders (the person in control of the motorcycle) appear to be somewhat more likely to die in an accident (rider / passenger fatality ratios where neither or both wore a helmet are both slightly above 1.0), we can see that when only one of the two people on the motorcycle wears a helmet, the person wearing the helmet is much less likely to die. Thus, hypothesis 1 (that helmets prevent deaths) is clearly supported.

The Florida newspaper article never makes a clear statement on whether helmets are actually effective; it draws no conclusions from its own dataset, and most of the article just quotes from various people on either side of the debate. The author seems to be doing the (now typical) "I'll stay out of it and just let pundits from either side say things" style of reporting. This is just as annoying here as it is with evolution vs. creationism reporting; when there are datasets of such high quality as the NHTSA's, it seems ridiculous to leave them out of an article to which they would add so much.

By using slipshod presentation, the Florida newspaper is making it just that much easier for anti-helmet advocates to argue their position. After all, many readers of this Florida paper will likely use these data to justify helmet laws, yet anti-helmet advocates can justifiably bring the entire analysis into question. Sometimes bad data analysis is worse than no data analysis at all.


References:

Deutermann, William. 2004. Motorcycle Helmet Effectiveness Revisited. Technical report prepared for the National Highway Traffic Safety Administration (NHTSA) by the National Center for Statistics and Analysis (NCSA). DOT HS 809 715 (HTML, PDF, entrance page)

Neale, Rick. 2006. Death rate soars for bikers: Player's crash drags debate over helmets back into the spotlight. Florida Today, June 18, 2006.

Categories:

• science
• statistics
• data
• analysis