Acupuncture and headaches

A while back I posted a link to an article attempting to show medical benefits of acupuncture for headache pain; in the post I left it up to Pharyngula and his links to discuss the study. The study compared patients in two groups: one where patients received the "standard treatment" and another where they received the "standard treatment" plus acupuncture. Pharyngula and others have commented that the study lacked a critical control group and thus was relatively meaningless. Apparently that wasn't good enough for one dedicated reader, so I'll take a stab at it here.

The comment the reader made was

"Ah, but it depends on the meaning of "control group". I've read more than a few pharmaceutical studies where the comparison is between some new drug and the usual "standard of care", rather than between the new drug and a placebo. That seems to be what's going on in this study, although I would have preferred that their non-control group received only acupuncture, rather than acupuncture+medication."

I see three primary differences between pharmaceutical studies and acupuncture studies that relate to this comment. First, pharmaceutical studies are often dealing with life and death conditions wherein the inclusion of a patient group receiving solely a placebo would be unethical if there is a treatment that has been shown to be effective. Second, many pharmaceutical treatments are perceived similarly from the patient's perspective (e.g. "OK, I'm now getting 3 blue pills a day instead of 3 red pills") and thus the possibility for placebo effects are relatively reduced, especially when we compare the difference in pharmaceutical treatments to the difference between poking someone in the head with needles and not poking someone in the head with needles. Third, many pharmaceutical studies can use non-subjective measures to determine the response to their treatments (e.g. white blood cell counts, pulse rate, respiration rate, body temperature, presence of infectious bacteria, etc.) which reduces the potential for confounding placebo effects. In this acupuncture study, however, the two groups of individuals were treated very differently, and the only measures that experimenters gathered were subjective.

A good analogy is for us to ponder the effectiveness of apples and oranges as projectiles. The acupuncture study is designed so that you're running a comparison of throwing an apple and an orange at one person while throwing just an apple at another person, then seeing how the two people react. The apple and orange person is likely to react more strongly, but is that because the orange is perceived as more dangerous, or because they had two projectiles thrown at them? We can't know. The better design would be to throw an apple and an orange at one person, while throwing two apples at another person. Now both people have two projectiles coming at them, and we can conclusively determine if oranges are different from apples. Or, of course, just throwing an apple at one person and an orange at another person would also work (as the commenter wisely states).

The key point is that the people in the groups to compare must be treated as equally as possible, with the treatment in question being the only difference. The primary missing control group in this study, from my perspective, is one wherein people believed that they were getting acupuncture (or some additional treatment), but were not actually receiving any acupuncture. This type of control group would allow us to determine whether the acupuncture itself is causing a physical change in the patient's headaches, or if simply telling a patient that you're treating their headache will make them feel better (a placebo effect). Never having had acupuncture I have no idea how hard it would be to provide sham acupuncture, but there are studies that have used sham acupuncture before, so it is possible.

It is reasonably likely that the study's results were biased by the placebo effect. Even though participants were randomized to a specific treatment (acupuncture vs. no acupuncture), all participants were informed about the study's methods and goals before the study began. Thus patients in the control group knew that they were not getting a treatment, while patients in the acupuncture group knew they were getting a treatment that others weren't; this is far from the ideal double-blind study. Since the study's measures were all subjective, participant knowledge of the experimental design could easily have skewed the results.

To be fair to the authors, I should point out that they address this issue in the discussion. They argue that other studies have shown that acupuncture is significantly different from sham treatments and other placebo effects with regard to pain management, and thus they weren't obligated to look for placebo effects. I e-mailed one of the authors (Dr. Andrew Vickers) to discuss this some, and he made the point that every study can't do everything at once, and that one should take the body of evidence as a whole and not just focus on one study (both good points). He sent me copies of a few papers that do compare acupuncture with placebo treatments for pain management, and I hope to post a summary of those in the coming weeks.

However, there are other problems with this acupuncture study besides the experimental design. In the methods they use one of the more minimalist descriptions of statistics I've seen, and throughout the results they do not include statistical information that should be present: there are no degrees of freedom, no F statistics, no regression coefficients. For comparison, I'm used to reading journals like Ecology, which have very specific statistical requirements.

It also appears that they have misinterpreted their statistical analyses. For instance take this portion of the results

"Table 4 shows data on use of resources. Patients in the acupuncture group made fewer visits to general practitioners and complementary practitioners than those not receiving acupuncture and took fewer days off sick."

and this portion of the discussion relating to those results

"We also found improvements in quality of life, decreases in use of medication and visits to general practitioners, and reductions in days off sick."

Those three sentences are their entire textual description of that data, excluding the table and the abstract. Based on the data in table 4, however, the P values for the three comparisons discussed in the results are all well above 0.05, and thus the differences are not statistically significant and should not be considered real (P=0.1 for visits to the general practitioner, 0.3 for complementary therapist, and 0.2 for days sick). Including false statements like these (implying there is an effect of acupuncture on certain measures when statistically there is none) is the type of data analysis that makes me skeptical about the rest of their study. [note: I could be interpreting their statistics incorrectly - I'd be surprised if a journal let this type of error slip through; if I am please let me know]

I want to make it clear that I'm not saying that acupuncture has no effect; I'm just saying that this study, by itself, doesn't show conclusively whether there is a real effect of acupuncture on headache pain. I also want to point out that this study only looked at the effect of acupuncture on headache pain, and thus has nothing to say about acupuncture's effectiveness (or lack thereof) on any other conditions.

The reference is: Vickers AJ, RW Rees, CE Zollman, R McCarney, CM Smith, N Ellis, P Fisher, and R Van Haselen. 2004. Acupuncture for chronic headache in primary care: large, pragmatic, randomised trial. British Medical Journal. 328: 744.