The Cochrane Collaboration 20 years later

Here is an editorial in BMJ on the history of the Cochrane Collaboration. It is not appropriately critical (overlooking a couple of the CC's huge blind spots) though it pretends to be. It ignores the fact that through the years the CC has been accommodating, even promotional, towards unscientific ideas like homeopathy and therapeutic touch. I documented that problem, with examples of reviews favorable, or neutral, to homeopathy and acupuncture as treatments for a variety of conditions, in this post. Unfortunately the links to the Cochrane reviews provided in that post are no longer active. But here is a Cochrane review stating that the only methodologically suitable trial showed a benefit of acupuncture in the treatment of mumps, and recommending that more trials be done. This review from the CC cited studies purporting to show a mortality and function benefit derived from acupuncture in the treatment of acute stroke, but said that methodologically stronger studies are needed. The list of these sorts of Cochrane “evidence based” reviews goes on and on. Apparently, in the philosophy of the CC, the biologic plausibility of a study doesn't matter in the least in critical appraisal of the results. Moreover, the nuttiness of an idea does not disqualify it as the topic for a review. What does qualify a topic for Cochrane review is popularity. If a modality of treatment is popular in some part of the world, no matter how implausible, it seems to warrant a review.

CC's other big blind spot is one I've commented on before and which has been termed by others as “methodolatry.” Definition:

the profane worship of the randomized clinical trial as the only valid method of investigation.

This is illustrated by a CC review of anticoagulation for acute venous thromboembolism (VTE):

Authors' conclusions

The limited evidence from RCTs of anticoagulants versus NSAIDs or placebo is inconclusive regarding the efficacy and safety of anticoagulants in VTE treatment. The use of anticoagulants is widely accepted in clinical practice, so a further RCT comparing anticoagulants to placebo could not ethically be carried out.

The authors' statements are a little weaselly. They're saying there's no good evidence to support anticoagulants but come short of recommending for or against them. And while saying it would not be ethical to do a RCT they come short of saying there is no equipoise. In fact, given their presentation of the evidence as showing unknown benefit they imply that there is equipoise.

These statements carry the potential for harm as illustrated by their citation on a web site purporting to advance the cause of evidence based medicine (EBM), the NNT, a popular open access site that encourages clinicians and patients to evaluate evidence for claims of popular treatments. The NNT's piece on anticoagulants for the treatment of VTE, using the Cochrane review as its only source material, states regarding the treatment of VTE with anticoagulants:

100% saw no benefit

0% were helped by being saved from death

Only through the lens of methodolatry could such a conclusion be reached. In fact there is evidence in favor of the treatment of VTE with anticoagulants. Though not from high quality placebo controlled RCTs the evidence is, in the aggregate, compelling.