Tuesday, February 19, 2008

Evidence based medicine does not equal science based medicine

That’s right, I said it. I’ve tried really hard over the last several years to learn about, read about and practice EBM. Along the way I’ve noticed something peculiar: evidence based medicine (EBM) has become a powerful enabler of complementary and alternative medicine (CAM). I touched on this briefly in a post in which I pondered the reasons why woo has so successfully corrupted academic medicine:

Finally, and somewhat paradoxically, the rise in unfounded CAM modalities may be an unintended consequence of the EBM movement. EBM advocates devalue scientific rationale and physiologic plausibility.

The problem may be a form of extreme empiricism applied to clinical questions. Empiricism, according to the dictionary, regards direct experience and observation as the only source of knowledge. In medicine, according to the definition, empiricism “disregards scientific theory and relies solely on practical experience.” That is not a core principle of EBM as I understand it, but it is a popular and pervasive distortion.

How many times, for example, have you heard the old saw “there is no alternative medicine; there is only medicine which has been shown to work and medicine which has not been shown to work?” That may be appealing at first glance, but EBM’s obsession with finding out what treatments “work”, with total disregard for biologic plausibility and prior knowledge, has spawned an explosion of dubious “clinical studies” on all sorts of woo, from acupuncture to those wooiest forms of woo such as homeopathy and Reiki. Seldom is anything solved by such studies. No claims are ever proven, nor are they totally dismissed. The lingering doubt perpetually fuels more inconclusive “research.” The track record of the National Center for Complementary and Alternative Medicine speaks for itself in this regard.

I examined some of the failings of EBM in a post last year. Methodologic flaws in CAM research, chance variation combined with positive publication bias and biased Medline indexing are just a few of the reasons. But I’m afraid I didn’t make these points nearly as well as Kimball Atwood recently did in two wonderful posts in the Science Based Medicine blog.

He points out that treatments must pass not only the evidentiary test but also the test of scientific plausibility. Because EBM devalues the latter it is inadequate for the evaluation of implausible claims even though it may perform well in evaluating plausible ones. This fundamental error is built into EBM’s system of analysis as illustrated by its evidence hierarchy, which places physiologic rationale and scientific principles at the bottom of the heap. Atwood illustrates the consequences of such faulty analysis in the first of his two posts:

Thus a “positive” clinical trial is given more weight than “physiology, bench research or ‘first principles’,” even when the latter definitively refute the claim.

Using the example of homeopathy, the focus of the first of his two posts, Atwood goes through a long list of fundamental scientific principles in opposition to the claims of homeopathy and then asks:

Is it realistic to assume that this “level” of evidence, when brought to bear on a claim that has no explanatory power in nature, can be overthrown by ambiguous clinical trials of dubious design? EBM tacitly makes that assumption.

In the second post Atwood gives the issue a more quantitative treatment by contrasting EBM’s use of popular frequentist statistics with Bayesian statistical analysis which seeks to determine how new evidence modifies prior knowledge (such as basic science principles, physiologic rationale, etc.). Most of us are familiar with the use of Bayesian analysis in evaluating laboratory test results in individual patients because it is widely advocated and taught as a tool for diagnosis. Although equally valid (and superior in many ways to the popular frequentist approach) for analysis of clinical trial data for evaluation of treatments, it is not a tool of EBM.

Atwood issues a plea for incorporation of Bayesian thinking in the evaluation of clinical claims because it takes into account scientific plausibility. According to Bayes’ theorem, whether you’re evaluating the probability of disease in a patient or the probability that a health claim is true the prior probability, P(A), based on what was known before, occupies the numerator of the equation. Thus if P(A) is zero no amount of observational data could establish the hypothesis as true. If P(A) is infinitesimally small nothing short of overwhelming experimental evidence could establish the hypothesis as true.

Homeopathy, Reiki and Therapeutic Touch immediately come to mind. Bayesian analysis of claims such as these is a formal and quantitative method of establishing what common sense has always told us concerning those occasional weakly positive, methodologically questionable studies of implausible claims. They’re what I call evidence based woo. Consider the miniscule value of P(A) in Bayes’ theorem for such claims as the woo factor, a factor which evidence based medicine leaves out of its analysis.

We need a balanced view. EBM proponents are correct in saying that pathophysiologic rationale alone is not sufficient. (They’re fond of trotting out the CAST study to make that point). But they are wrong to ignore and devalue such knowledge. In order to evaluate the claims of CAM we need clinical evidence taken in the light of fundamental biologic principles.


james gaulte said...

I had intended to author a post with those same themes and may yet.As is so often the case you got there first.Great post and it and those posts on this subject on Science-Based Medicine deserve to handed out to medical students classes.May then, fewer of them would go ga-ga over woo.

Robert W Donnell said...

I hope you write the post. The more voices who join the chorus, the better.

Dana Ullman, MPH said...

I am intrigued by arm-chair philosophers (non-scientists) who assert that some therapies in CAM are "implausible." I personally am more interested in evidence, research, and results than what some people's worldviews limit their thinking.

I could help but notice your distain for homeopathy as well as your recent blogs on sepsis. I therefore could not help but tie these subjects together...in the light of evidence-based medicine (and evidence-based science).

At a University of Vienna hospital, 70 patients with severe sepsis were enrolled in a randomized double-blind, placebo-controlled clinical trial, measuring survival rates at 30 days and at 180 days (Frass, et al. 2005). Those patients given a homeopathic medicine were prescribed it in the 200C potency only (in 12 hour intervals during their hospital stay). The most frequently prescribed medicines were: Apis mellifica, Arsenicum album, Baptisia, Bryonia, Carbo vegetabilis, Crotalus horridus, Lachesis muta*, Lycopodium clavatum, Phosphorus*, and Pyrogenium (those with an “*” were prescribed more frequently than other medicines).

The survival rate at day 30 was 81.8% for homeopathic patients and 67.7% for those given a placebo. At day 180, 75.8% of homeopathic patients survived and only 50.0% of the placebo patients survived (p=0.043). One patient was saved for every four who were treated.
The researchers concluded that homeopathy may be a useful addition with long-term benefit for severely septic patients. A constraint is limited number of homeopaths available in hospital settings.


Frass M, Linkesch, M, Banjya, S, et al. Adjunctive homeopathic treatment in patients with severe sepsis: a randomized, double-blind, placebo-controlled trial in an intensive care unit. Homeopathy 2005:94;75–80

Martin said...

Interesting to see, that CAM-RCTs are of higher methodol. quality that "conventional-medicines" RCTs: http://www.ncbi.nlm.nih.gov/pubmed/16018912

Anonymous said...

Yours and KA's posts on this are very helpful.

Over at SBM I suggested a valuable worked example for a number of people might be estimating the prior probability for Marshall and Warren’s early work on Helicobacter pylori and its impact on gastroduodenal management. I frequently have Marshall quoted to me as a variation on the Galileo gambit so establishing whether he and Warren would have been helped or hindered by Bayesian techniques would be useful.

I can see that Marshall and Warren will have had physiology and bench work on their side but what would others have made of the plausibility back in their earliest days?

Anonymous said...

That's a very interesting post.

Anonymous said...

I assume Dana is a proponent of homeopathy and I am sure we have discussed this issue in another blog post (unless there are two Dana Ullmans!). I also remember that you never responded to my rebuttal of your references! You just post something dubious and then disappear.

In this case, you seem to not have read Dr. RW post at all! Your post of a single trial with non conclusive results as support for homeopathy shows exactly that.

You are also saying that you don't like arm-chair philosophers calling on the implausibility of homeopathy, yet you do not defend homeopathy with plausible arguments -which makes it for a strong ad-hominem.

Of course you cannot provide plausible arguments because homeopathy *is* in fact totally implausible by the current scientific knowledge. "Like cures like", "higher dilutions are stronger" etc. Let's see if there will be any reasonable response this time.

Anonymous said...

It is very difficult to study Bayesian Analysis topic. Not many good reference textbooks to study Markov chain.

I use Statistical Decision Theory and Bayesian Analysis, 2nd Edition to study. This is good reference textbook.

Do you have any other good Bayesian Analysis related textbooks recommend?


Andy ^_^
Cocomartini Discount Online Bookstore

Lenoxus said...

I know this thread has been zombiefied by time, but I'm confused enough to post anyway.

Why I'm confused is this: the post seems to be saying that EBM has been kind to homeopathy, in terms of the results. My understanding is that it hasn't; study after study (DBRT, that is) has shown homeopathy to be no better than placebo, and the only things homeopaths can muster in response is "The treatments weren't individualized properly" (or "Well, look at this one study where it beat the margin of error by 1%!").

Homeopathy is not something that is chemically implausible yet evidentially shown to work anyway. It's simply a placebo from beginning to end, no?

(I got to this page from a Google search for the phrase "evidence based medicine", which I'd first encountered as a pejorative from homeopaths — you know, "You always think everything real comes down to evidence, hmmph".)