By REED ABELSON and GARDINER HARRIS
The New York Times
June 2, 2010
In selling the health care overhaul to Congress, the Obama administration cited a once obscure research group at Dartmouth College to claim that it could not only cut billions in wasteful health care spending but make people healthier by doing so.
Wasteful spending — perhaps $700 billion a year — “does nothing to improve patient health but subjects you and me to tests and procedures that aren’t necessary and are potentially harmful,” the president’s budget director, Peter Orszag, wrote in a blog post characteristic of the administration’s argument.
Mr. Orszag even displayed maps produced by Dartmouth researchers that appeared to show where the waste in the system could be found. Beige meant hospitals and regions that offered good, efficient care; chocolate meant bad and inefficient.
The maps made reform seem relatively easy to many in Congress, some of whom demanded the administration simply trim the money Medicare pays to hospitals and doctors in the brown zones. The administration promised to seriously consider doing just that.
But while the research compiled in the Dartmouth Atlas of Health Care has been widely interpreted as showing the country’s best and worst care, the Dartmouth researchers themselves acknowledged in interviews that in fact it mainly shows the varying costs of care in the government’s Medicare program. Measures of the quality of care are not part of the formula.
For all anyone knows, patients could be dying in far greater numbers in hospitals in the beige regions than hospitals in the brown ones, and Dartmouth’s maps would not pick up that difference. As any shopper knows, cheaper does not always mean better.
Even Dartmouth’s claims about which hospitals and regions are cheapest may be suspect. The principal argument behind Dartmouth’s research is that doctors in the Upper Midwest offer consistently better and cheaper care than their counterparts in the South and in big cities, and if Southern and urban doctors would be less greedy and act more like ones in Minnesota, the country would be both healthier and wealthier.
But the real difference in costs between, say, Houston and Bismarck, N.D., may result less from how doctors work than from how patients live. Houstonians may simply be sicker and poorer than their Bismarck counterparts. Also, nurses in Houston tend to be paid more than those in North Dakota because the cost of living is higher in Houston. Neither patients’ health nor differences in prices are fully considered by the Dartmouth Atlas.
The mistaken belief that the Dartmouth research proves that cheaper care is better care is widespread — and has been fed in part by Dartmouth researchers themselves.
The debate about the Dartmouth work is important because a growing number of health policy researchers are finding that overhauling the nation’s health care system will be far harder and more painful than the Dartmouth work has long suggested. Cuts, if not made carefully, could cost lives.
Looking in detail at Dartmouth’s evidence helps show why.
Criticisms on Spending
A main focus of the Dartmouth Atlas is comparing spending among the nation’s hospitals. To do that, Dartmouth researchers use data on how much hospitals have billed Medicare for patients with a chronic illness who were in their last six months or two years of life.
“We show where the waste is in medicine,” said Dr. Elliott Fisher, a physician who is one of the principal authors of the Dartmouth work and was a frequent visitor to Washington during the long legislative debate. “If everyone could operate like Oregon, Seattle or the Upper Midwest, there’s huge savings.”
But the atlas’s hospital rankings do not take into account care that prolongs or improves lives. If one hospital spends a lot on five patients and manages to keep four of them alive, while another spends less on each but all five die, the hospital that saved patients could rank lower because Dartmouth compares only costs before death.
“It may be that some places that are spending more are actually getting better results,” said Dr. Harlan M. Krumholz, a professor of medicine and health policy expert at Yale.
Failing to receive credit for better care enrages some hospital administrators. But for the Dartmouth researchers, making these administrators uncomfortable is the point of the rankings.
“When you name names, people start paying more attention,” Dr. Fisher said. “We never asserted and never claimed that we judged the quality of care at a hospital — only the cost.”
In interviews, administration officials acknowledged that the Dartmouth Atlas was far from perfect. Mr. Orszag says he does not rely on the atlas alone to prove that huge savings are possible.
“What I have repeatedly said is that a wide variety of evidence suggests there is substantial opportunity for savings, and the challenge is in capturing that opportunity,” he said.
Still, the Dartmouth work remains influential in Washington.
Dr. Donald Berwick, nominated by President Obama to run Medicare, called it the most important research of its kind in the last quarter-century. In March, in response to the Congressional Democrats who would have otherwise withheld their support for the health legislation, the administration made a promise. It said it would ask the Institute of Medicine, a nongovernment advisory group, to consider ways of putting the Dartmouth findings into action by setting payment rates that would punish inefficient hospitals and reward efficient ones.
But if that system penalizes big city hospitals like those at the Ronald Reagan UCLA Medical Center and NYU Langone Medical Center — which look profligate by Dartmouth’s measure but may rank much higher by other quality indicators — a battle over the validity of the Dartmouth work is almost certain in Congress.
In fact, among health policy analysts, that battle has already begun. Critiques have been published in prominent medical journals, and more are on the way.
Differences in Patient Care
In interviews and extensive written comments, Dr. Fisher and Jonathan Skinner, a health economist at Dartmouth who works with him, vigorously defended their work. They berated what they described as “a cottage industry in new studies trying to debunk our findings.”
They say their critics fail to understand the issues and often make significant statistical errors. And they say even if they adjusted more fully to reflect differences in regional costs and patients’ health, the overall effect on the atlas’s findings would be relatively small.
The researchers also say they have made some of those adjustments in some of their other published work. Many other health researchers say Dartmouth should be praised for highlighting the tremendous differences in how patients are treated and for emphasizing that patients often fail to benefit from additional care.
“Dartmouth opened our eyes to something that I believe is real and important,” said Dr. Robert M. Wachter, a professor in hospital medicine at the University of California, San Francisco.
But even those who defend Dartmouth say that failing to make basic data adjustments undermines the geographic variations the atlas purports to show. David Cutler, a professor of economics at Harvard, likens it to failing to account for inflation when looking at gross domestic product. “Nobody in their right mind would talk about G.D.P. growth without adjusting for prices,” he said.
In addition to their hospital rankings, the Dartmouth researchers have also done separate studies of how Medicare spending affects patient care regionally. A 2003 study found tha
t patients who lived in places most expensive for the Medicare program received no better care than those who lived in cheaper areas.
Because some regions spent nearly a third more than other regions without any apparent benefit, the Dartmouth team concluded that at least one dollar in three was wasted by Medicare. When applied generally to the nation’s health care system, that meant about $700 billion could be saved.
But as it began publicly discussing its research, the Dartmouth team often extrapolated beyond this basic finding. Not only do high-spending regions fail to provide better care, the Dartmouth team began to argue, but those regions actually offer worse care.
In just one example of this extrapolation, Dr. Fisher, in testimony before Congress last year, summarized his and others’ work by asking, “Why are access and quality worse in high-spending regions?”
And on Dartmouth’s Web site, a question-and-answer section suggests that this interpretation is appropriate:
“The evidence is that higher utilization does not extend life expectancy, and might be correlated with shorter life expectancy, compared with lower utilization. Therefore, sending people with chronic diseases to higher-efficiency, lower-utilization hospitals for their care could result in both lower spending and increased quality and length of life.”
While a few studies by other researchers have shown that more spending leads to worse health, some others have suggested the opposite — that more expensive hospitals might offer better care. But many have shown no link, either way, between spending and quality.
In other words, there is little evidence to support the widely held view, shaped by the Dartmouth researchers, that the nation’s best hospitals tend to be among the least expensive.
In interviews, Dr. Fisher and Mr. Skinner acknowledged that there was no proven link between greater spending and worse health outcomes. And Dr. Fisher acknowledged the apparent inconsistency between his statements in interviews with The New York Times and those made elsewhere, saying that he was sometimes less careful in discussing his team’s research than he should be.
In any case, the more-is-worse message has resonated with insurers, whose foundations now help to finance the Dartmouth Atlas. Dartmouth researchers also created a company, Health Dialog, to consult for insurers and others on Dartmouth’s findings. Valued at nearly $800 million, the company was sold to a British insurer in 2007 and still helps to finance the Dartmouth work.
Rankings in Wisconsin
Last June, as Mr. Obama campaigned for his health care overhaul, he visited Green Bay, Wis., praising the city for getting “more quality out of fewer health care dollars than many other communities.”
Two of Green Bay’s hospitals, Bellin and St. Mary’s Hospital Medical Center, rank fourth and 11th within Wisconsin on the Dartmouth list.
But again, Dartmouth ranks hospitals only by costs and number of treatments and procedures. A different picture emerges from work done by the Wisconsin Collaborative for Healthcare Quality, a voluntary group of health care organizations that uses both price and quality of care measures. In an analysis of heart attack care, for example, it ranks Bellin second, and St. Mary’s 15th, among the 22 hospitals in the state.
And a Medicare ranking based on its own data that shows how many people die after treatment for certain conditions — statistics that exclude costs entirely — puts Bellin fifth, but drops St. Mary’s to second-to-last: 67th of the 68 hospitals statewide that were measured by both Dartmouth and Medicare.
Do the Green Bay hospitals favored by Dartmouth really offer better care? Maybe not.
Similar problems arise with Dartmouth’s regional data. In Dartmouth’s rankings, for instance, New Jersey comes in dead last because its costs per Medicare beneficiary are the nation’s highest. And yet, for the quality of care offered in New Jersey, independent of cost, federal health officials rank New Jersey second only to Vermont.
Researchers who have examined the Dartmouth Atlas numbers have found other flaws that can distort hospital rankings. Doctors at Cedars-Sinai in Los Angeles, for instance, found that Dartmouth had failed to distinguish two different types of intensive care units the hospital runs. Dartmouth also might have overcounted the number of specialists examining each patient.
The errors, Cedars-Sinai said, meant the hospital probably fared worse in the Dartmouth rankings than it should have. Cedars-Sinai’s chief medical officer, Dr. Michael Langberg, said he had tried to discuss these problems with Dartmouth researchers but had thought they were unresponsive.
Dr. Fisher said he had tried to address Cedars-Sinai’s concerns. But he also argued that while the Dartmouth Atlas was not perfect, it provided useful insights into differences in the way various hospitals treat patients.
“For the past 20 years, my colleagues and I and now many others have been working hard to clarify the causes and consequences of regional variations in practice and spending,” Dr. Fisher said. “The work is challenging and more work needs to be done, but we have learned enough to help guide health reform.”