Evidence based medicine does not equal science based medicine

That’s right, I said it. I’ve tried really hard over the last several years to learn about, read about and practice EBM. Along the way I’ve noticed something peculiar: evidence based medicine (EBM) has become a powerful enabler of complementary and alternative medicine (CAM). I touched on this briefly in a post in which I pondered the reasons why woo has so successfully corrupted academic medicine:

Finally, and somewhat paradoxically, the rise in unfounded CAM modalities may be an unintended consequence of the EBM movement. EBM advocates devalue scientific rationale and physiologic plausibility.

The problem may be a form of extreme empiricism applied to clinical questions. Empiricism, according to the dictionary, regards direct experience and observation as the only source of knowledge. In medicine, according to the definition, empiricism “disregards scientific theory and relies solely on practical experience.” That is not a core principle of EBM as I understand it, but it is a popular and pervasive distortion.

How many times, for example, have you heard the old saw “there is no alternative medicine; there is only medicine which has been shown to work and medicine which has not been shown to work?” That may be appealing at first glance, but EBM’s obsession with finding out what treatments “work”, with total disregard for biologic plausibility and prior knowledge, has spawned an explosion of dubious “clinical studies” on all sorts of woo, from acupuncture to those wooiest forms of woo such as homeopathy and Reiki. Seldom is anything solved by such studies. No claims are ever proven, nor are they totally dismissed. The lingering doubt perpetually fuels more inconclusive “research.” The track record of the National Center for Complementary and Alternative Medicine speaks for itself in this regard.

I examined some of the failings of EBM in a post last year. Methodologic flaws in CAM research, chance variation combined with positive publication bias and biased Medline indexing are just a few of the reasons. But I’m afraid I didn’t make these points nearly as well as Kimball Atwood recently did in two wonderful posts in the Science Based Medicine blog.

He points out that treatments must pass not only the evidentiary test but also the test of scientific plausibility. Because EBM devalues the latter it is inadequate for the evaluation of implausible claims even though it may perform well in evaluating plausible ones. This fundamental error is built into EBM’s system of analysis as illustrated by its evidence hierarchy, which places physiologic rationale and scientific principles at the bottom of the heap. Atwood illustrates the consequences of such faulty analysis in the first of his two posts:

Thus a “positive” clinical trial is given more weight than “physiology, bench research or ‘first principles’,” even when the latter definitively refute the claim.

Using the example of homeopathy, the focus of the first of his two posts, Atwood goes through a long list of fundamental scientific principles in opposition to the claims of homeopathy and then asks:

Is it realistic to assume that this “level” of evidence, when brought to bear on a claim that has no explanatory power in nature, can be overthrown by ambiguous clinical trials of dubious design? EBM tacitly makes that assumption.

In the second post Atwood gives the issue a more quantitative treatment by contrasting EBM’s use of popular frequentist statistics with Bayesian statistical analysis which seeks to determine how new evidence modifies prior knowledge (such as basic science principles, physiologic rationale, etc.). Most of us are familiar with the use of Bayesian analysis in evaluating laboratory test results in individual patients because it is widely advocated and taught as a tool for diagnosis. Although equally valid (and superior in many ways to the popular frequentist approach) for analysis of clinical trial data for evaluation of treatments, it is not a tool of EBM.

Atwood issues a plea for incorporation of Bayesian thinking in the evaluation of clinical claims because it takes into account scientific plausibility. According to Bayes’ theorem, whether you’re evaluating the probability of disease in a patient or the probability that a health claim is true the prior probability, P(A), based on what was known before, occupies the numerator of the equation. Thus if P(A) is zero no amount of observational data could establish the hypothesis as true. If P(A) is infinitesimally small nothing short of overwhelming experimental evidence could establish the hypothesis as true.

Homeopathy, Reiki and Therapeutic Touch immediately come to mind. Bayesian analysis of claims such as these is a formal and quantitative method of establishing what common sense has always told us concerning those occasional weakly positive, methodologically questionable studies of implausible claims. They’re what I call evidence based woo. Consider the miniscule value of P(A) in Bayes’ theorem for such claims as the woo factor, a factor which evidence based medicine leaves out of its analysis.

We need a balanced view. EBM proponents are correct in saying that pathophysiologic rationale alone is not sufficient. (They’re fond of trotting out the CAST study to make that point). But they are wrong to ignore and devalue such knowledge. In order to evaluate the claims of CAM we need clinical evidence taken in the light of fundamental biologic principles.