One of the site’s creators, Graham Walker, chief emergency resident at St. Luke’s-Roosevelt Hospital Center in New York City, tells the Health Blog he was surprised that data on the effectiveness of treatments “was out there, but not easily accessible.”
Well that's odd sounding. Graham Walker's a pretty smart guy and I suspect EBM's nothing new to him. Did he really not know the data were out there? Or if he did, has he just decided it's not easily accessible? (It's been easily accessible for some time, of course).
If the NNT is critically appraising evidence who's critically appraising The NNT? Certainly not the WSJ Health Blog. There are some very concerning questions about the content that need to be asked. I'll attempt to address some of them here, but first a general comment. The basic idea of the project is sound. The process of EBM, which involves formulating the clinical question, hunting for evidence, critically appraising the evidence, assessing the magnitude of treatment effect and then applying all that to your patient's unique attributes is time consuming as I once blogged here and here. There are some secondary evidence sources on the web that perform some of those steps for you so you can take short cuts. The NNT does this in a very direct and simple way.
But maybe it's too simple. There are serious problems with the site's content. Let's look at a few examples. The section on TPA for ischemic stroke concludes “no benefits have been found for this intervention” and states that 1 in 20 patients will be harmed and none will benefit. A casual reader might wonder why the FDA hasn't yanked its approval, but those readers familiar with the long running controversy about TPA for ischemic stroke know where this is coming from. It becomes even more clear when you realize an important source of bias: all the site authors are emergency medicine physicians. It's well known that emergency medicine has been pushing back against TPA for ischemic stroke for years---almost ever since its approval for the indication. A balanced discussion of the controversy would have been welcome. Instead the authors seem to have spun the literature to the negative side in every conceivable way. I have blogged extensively about the nuances of this issue before and my take has been that while I am concerned about the consequences of declaring it as standard of care for ischemic stroke, I believe that, when applied with careful observance of the protocol and exclusion criteria, with appropriate systems of care in place and with meaningful informed consent, it is a useful intervention.
So how did they spin the evidence? First, in their evidence synthesis they included trials (9 out of the 11 they looked at) with a longer time window than that for which the drug is used. They justify that with this weak argument:
It has been argued that the difference between studies suggesting benefit and all other studies is the timing of the intervention. The Cochrane findings do not support this view, nor does one extensive post-hoc analysis.13 More importantly, the most recent study (ECASS-3) suggested benefits when thrombolytics were administered between 3 and 4.5 hours. This is outside the window that purportedly distinguished the NINDS trials, and effectively neutralizes time as the defining factor that separates trials concluding benefit from those that do not.
That issue is best addressed by this Lancet meta-analysis from May of this year:
Findings
Treatment was started within 360 min of stroke onset in 3670 patients randomly allocated to alteplase (n=1850) or to placebo (n=1820). Odds of a favourable 3-month outcome increased as OTT decreased (p=0·0269) and no benefit of alteplase treatment was seen after around 270 min. Adjusted odds of a favourable 3-month outcome were 2·55 (95% CI 1·44–4·52) for 0–90 min, 1·64 (1·12–2·40) for 91–180 min, 1·34 (1·06–1·68) for 181–270 min, and 1·22 (0·92–1·61) for 271–360 min in favour of the alteplase group. Large parenchymal haemorrhage was seen in 96 (5·2%) of 1850 patients assigned to alteplase and 18 (1·0%) of 1820 controls, with no clear relation to OTT (p=0·4140). Adjusted odds of mortality increased with OTT (p=0·0444) and were 0·78 (0·41–1·48) for 0–90 min, 1·13 (0·70–1·82) for 91–180 min, 1·22 (0·87–1·71) for 181–270 min, and 1·49 (1·00–2·21) for 271–360 min.
Interpretation
Patients with ischaemic stroke selected by clinical symptoms and CT benefit from intravenous alteplase when treated up to 4·5 h. To increase benefit to a maximum, every effort should be taken to shorten delay in initiation of treatment. Beyond 4·5 h, risk might outweigh benefit.
Although the NNT site updated the section on TPA in August of 2010 it contains no mention of the May meta-analysis.
In addition, the authors claimed that NINDS 1 showed no benefit from TPA. That conclusion is distorted. Although NINDS 1 did not show benefit at 24 hours NINDS 1, like NINDS 2, showed benefit at three months.
Also concerning is their analysis of anticoagulation for VTE. They conclude that while a few will be harmed by bleeding, 100% of patients will derive no benefit. It's hard to know where they're coming from here. There's no reason I can think of that they'd be biased against anticoagulation. Maybe they worship at the throne of the Cochrane database. The Cochrane review they cite indicates that there have been only two trials in recent history that address this issue, both of which were underpowered to detect a difference in outcomes. The lead author of that review, by the way, has a remarkable conflict of interest. Maybe they are approaching the topic from an extreme EBM purist view. From that point of view one could say, very technically speaking, that anticoagulation for VTE is not “evidence based.” I dealt with that issue in posts here and here. (BMJ's parachute study comes to mind). It's an issue that deserves a more balanced approach than the NNT authors gave it. For a properly nuanced analysis, that makes the case that even though anticoagulation for VTE is not technically “evidence based” the evidence favoring treatment is overwhelming, the full text of this paper is a must read. (Also see my aforementioned posts for a description of the remarkable COI of the lead author of the Cochrane review). I don't want my next patient presenting with VTE to refuse anticoagulation. I hope they haven't read that section of NNT.
I have some concerns about other sections of the site, which I may address in a future post.
2 comments:
Thanks for the feedback!
I disagree that the data is easily accessible--if that's the case, then why do so many guidelines, references, and standards continue to ignore it so egregiously? Why are so many falsehoods so well propagated through medical school, residency, and fellowship?
We're clearly not going to settle the TPA debate here--but I will say this:
* the Lancet review you cite includes several trials (ATLANTIS) that were stopped early for harm;
* I guess there's something special about TPA, because other trials using streptokinase were excluded;
* Multiple authors of the Lancet meta-analysis cite their own conflicts of interest in the article, and, to be fair, if Emergency Physicians are anti-TPA, then neurologists are pro-TPA;
* There are so so so many sketchy things about how the FDA approved TPA that I personally have a problem with a lot of the data from the outright.
* NINDS-1's primary outcome was improvement at 24 hours, not at 3 months. It did not show a benefit, which is what people often say: "Look, he just got TPA, and he's already moving his arm!"
As far as anticoagulation goes, we state that "Large, properly designed trials are needed." Also, I didn't know that about the Cochrane author. Thanks. I personally think that anticoagulation probably does work--but we're trying to discuss the evidence available on our site, not what I want to believe. I like the JIM article you linked, but I have problems believing compiled case series or lists of studies where patients had a "clinical PE" but no official diagnosis. (I like the PIOPED fallout group data.)
I think it's pretty remarkable, and worth letting the public know that we have no good randomized trial data for medicines that have definite significant side effects (and maybe significant benefits).
Thanks. Very thoughtful and thought provoking comments.
I actually think most (not all) guidelines get it right. But as an example of what you're talking about we could look at anticoagulation for VTE as presented on NNT and compare that with the ACCP guidelines. I think the ACCP guideline authors are just following the principle of EBM that says you use the highest level of evidence you have, knowing that that may not be a RCT. All the evidence for anticoagulation is "lower level" but in the aggregate it's very convincing. Moreover it's the best we're going to get. As far as large properly designed trials being needed, I don't think a placebo controlled trial of anticoagulation for VTE would be ethical, do you?
And although it is certainly of interest to point out that anticoagulation for VTE was never proven with such high level evidence (as I did in the background posts I linked) that's something for the discerning reader. I have to wonder if the consumer would read it and get the wrong message. Most wouldn't understand the nuances.
I agree we can leave TPA aside. Concerning your comment about Emergency Physicians and Neurologists, yes we all have our biases. Sorting through it all is part of what makes medicine fun and interesting.
Post a Comment