One of the site’s creators, Graham Walker, chief emergency resident at St. Luke’s-Roosevelt Hospital Center in New York City, tells the Health Blog he was surprised that data on the effectiveness of treatments “was out there, but not easily accessible.”
Well that's odd sounding. Graham Walker's a pretty smart guy and I suspect EBM's nothing new to him. Did he really not know the data were out there? Or if he did, has he just decided it's not easily accessible? (It's been easily accessible for some time, of course).
If the NNT is critically appraising evidence who's critically appraising The NNT? Certainly not the WSJ Health Blog. There are some very concerning questions about the content that need to be asked. I'll attempt to address some of them here, but first a general comment. The basic idea of the project is sound. The process of EBM, which involves formulating the clinical question, hunting for evidence, critically appraising the evidence, assessing the magnitude of treatment effect and then applying all that to your patient's unique attributes is time consuming as I once blogged here and here. There are some secondary evidence sources on the web that perform some of those steps for you so you can take short cuts. The NNT does this in a very direct and simple way.
But maybe it's too simple. There are serious problems with the site's content. Let's look at a few examples. The section on TPA for ischemic stroke concludes “no benefits have been found for this intervention” and states that 1 in 20 patients will be harmed and none will benefit. A casual reader might wonder why the FDA hasn't yanked its approval, but those readers familiar with the long running controversy about TPA for ischemic stroke know where this is coming from. It becomes even more clear when you realize an important source of bias: all the site authors are emergency medicine physicians. It's well known that emergency medicine has been pushing back against TPA for ischemic stroke for years---almost ever since its approval for the indication. A balanced discussion of the controversy would have been welcome. Instead the authors seem to have spun the literature to the negative side in every conceivable way. I have blogged extensively about the nuances of this issue before and my take has been that while I am concerned about the consequences of declaring it as standard of care for ischemic stroke, I believe that, when applied with careful observance of the protocol and exclusion criteria, with appropriate systems of care in place and with meaningful informed consent, it is a useful intervention.
So how did they spin the evidence? First, in their evidence synthesis they included trials (9 out of the 11 they looked at) with a longer time window than that for which the drug is used. They justify that with this weak argument:
It has been argued that the difference between studies suggesting benefit and all other studies is the timing of the intervention. The Cochrane findings do not support this view, nor does one extensive post-hoc analysis.13 More importantly, the most recent study (ECASS-3) suggested benefits when thrombolytics were administered between 3 and 4.5 hours. This is outside the window that purportedly distinguished the NINDS trials, and effectively neutralizes time as the defining factor that separates trials concluding benefit from those that do not.
That issue is best addressed by this Lancet meta-analysis from May of this year:
Treatment was started within 360 min of stroke onset in 3670 patients randomly allocated to alteplase (n=1850) or to placebo (n=1820). Odds of a favourable 3-month outcome increased as OTT decreased (p=0·0269) and no benefit of alteplase treatment was seen after around 270 min. Adjusted odds of a favourable 3-month outcome were 2·55 (95% CI 1·44–4·52) for 0–90 min, 1·64 (1·12–2·40) for 91–180 min, 1·34 (1·06–1·68) for 181–270 min, and 1·22 (0·92–1·61) for 271–360 min in favour of the alteplase group. Large parenchymal haemorrhage was seen in 96 (5·2%) of 1850 patients assigned to alteplase and 18 (1·0%) of 1820 controls, with no clear relation to OTT (p=0·4140). Adjusted odds of mortality increased with OTT (p=0·0444) and were 0·78 (0·41–1·48) for 0–90 min, 1·13 (0·70–1·82) for 91–180 min, 1·22 (0·87–1·71) for 181–270 min, and 1·49 (1·00–2·21) for 271–360 min.
Patients with ischaemic stroke selected by clinical symptoms and CT benefit from intravenous alteplase when treated up to 4·5 h. To increase benefit to a maximum, every effort should be taken to shorten delay in initiation of treatment. Beyond 4·5 h, risk might outweigh benefit.
Although the NNT site updated the section on TPA in August of 2010 it contains no mention of the May meta-analysis.
In addition, the authors claimed that NINDS 1 showed no benefit from TPA. That conclusion is distorted. Although NINDS 1 did not show benefit at 24 hours NINDS 1, like NINDS 2, showed benefit at three months.
Also concerning is their analysis of anticoagulation for VTE. They conclude that while a few will be harmed by bleeding, 100% of patients will derive no benefit. It's hard to know where they're coming from here. There's no reason I can think of that they'd be biased against anticoagulation. Maybe they worship at the throne of the Cochrane database. The Cochrane review they cite indicates that there have been only two trials in recent history that address this issue, both of which were underpowered to detect a difference in outcomes. The lead author of that review, by the way, has a remarkable conflict of interest. Maybe they are approaching the topic from an extreme EBM purist view. From that point of view one could say, very technically speaking, that anticoagulation for VTE is not “evidence based.” I dealt with that issue in posts here and here. (BMJ's parachute study comes to mind). It's an issue that deserves a more balanced approach than the NNT authors gave it. For a properly nuanced analysis, that makes the case that even though anticoagulation for VTE is not technically “evidence based” the evidence favoring treatment is overwhelming, the full text of this paper is a must read. (Also see my aforementioned posts for a description of the remarkable COI of the lead author of the Cochrane review). I don't want my next patient presenting with VTE to refuse anticoagulation. I hope they haven't read that section of NNT.
I have some concerns about other sections of the site, which I may address in a future post.