Sunday, June 06, 2010

The Dartmouth Atlas project: spun too far!

By now you’re probably aware of the controversial New York Times article which questions popular conclusions drawn from the fabled Dartmouth Atlas project. According to the Times piece:

Mr. Orszag even displayed maps produced by Dartmouth researchers that appeared to show where the waste in the system could be found. Beige meant hospitals and regions that offered good, efficient care; chocolate meant bad and inefficient.

The maps made reform seem relatively easy to many in Congress, some of whom demanded the administration simply trim the money Medicare pays to hospitals and doctors in the brown zones. The administration promised to seriously consider doing just that.

The point being that the Dartmouth Atlas data may have led to false optimism about out ability to fund health care by merely reducing waste.

It goes on:

But while the research compiled in the Dartmouth Atlas of Health Care has been widely interpreted as showing the country’s best and worst care, the Dartmouth researchers themselves acknowledged in interviews that in fact it mainly shows the varying costs of care in the government’s Medicare program. Measures of the quality of care are not part of the formula.

For all anyone knows, patients could be dying in far greater numbers in hospitals in the beige regions than hospitals in the brown ones, and Dartmouth’s maps would not pick up that difference. As any shopper knows, cheaper does not always mean better.

Even Dartmouth’s claims about which hospitals and regions are cheapest may be suspect.

And so on.

The article drew some criticism in the blogs and an angry response from the Dartmouth Atlas guys themselves. Indeed the article contains some distortions which is unfortunate because it has a point to make. The popular spin is that the variation in health care costs revealed by the Dartmouth data is simply a result of waste and greed. The best health care systems, according to the logic, are those with the lowest expenditures. That’s based on a faulty assumption: that the outcomes in all regions are similar, or better in the regions with lower expenditures. The evidence in support of that premise is scant and very soft, coming from an old study of selected outcomes using early 1990s data. Cost estimates for different regions were drawn entirely from terminal patients. That was the investigators’ way of eliminating severity of illness as the cause of cost variation.

In response to the Times article the Dartmouth investigators countered that their maps do take into account quality of care (see this map). The trouble is, their surrogate for quality, the CMS performance measures, is irrelevant. Only readers naïve enough to believe that CMS and JC performance measures reflect real quality would be convinced by those data.

The Dartmouth Atlas was spun far beyond what the evidence supported, for political ends. That’s where the problem lies, not with the project itself. The data need to be viewed within the limitations of the methods. The findings are relevant. The sheer enormity of the variation in cost says deviation from best practice is widespread. Many questions remain unanswered. One is why? What external factors drive the variation? Another is in what direction? The popular assumption is that the error is in the direction of over utilization but it could just as easily be the other way around. After all, that’s what the best quality data we have and a sampling of public opinion say.

Other selected blog reactions:

DB said:

Higher spending is not necessarily bad. We rather must examine what we are buying for that money. Sometimes money helps.

Wesley Smith at Secondhand Smoke agreed with me:

Don’t get me wrong, I am not criticizing the study. I am criticizing the government and Obamacare boosters for misrepresenting it.

No comments: