Observational intensity bias associated with illness adjustment: cross sectional analysis of insurance claims

By John E Wennberg, Douglas O Staiger, Sandra M Sharp, Daniel J Gottlieb, Gwyn Bevan, Klim McPherson, H Gilbert Welch
BMJ, February 21, 2013


We have shown that a method of risk adjustment that used data on diagnoses and controlled for the effects of supply, by using data on the frequency of visits by physicians in the year prior to a patient’s death, was more efficient than the standard method; but that still accounted for less than 25% of geographic variation in age, sex, and race adjusted mortality among fee for service Medicare beneficiaries. Thus, our study points to the importance of developing risk adjustment methods that better explain variation in age, sex, and race mortality rates and suggests that these will be found by using data that are clearly independent of the effects of supply.



Dartmouth Study Questions Widely Used Risk-Adjustment Methods

By Jordan Rau
Kaiser Health News, February 21, 2013

In evaluating a hospital and health plan in the increasingly expensive U.S. health care system, federal officials and researchers often first factor in an assessment of how sick their patients are. A new study, however, challenges the validity of several widely used “risk-adjustment” efforts and suggests that Medicare is overpaying some plans and facilities while underpaying others.

Without these risk adjustments to level the comparisons, a hospital with more frail and very ill patients—who are more likely to die — might incorrectly appear to be doing a worse job than a hospital with healthier patients — who are more likely to survive.

Medicare risk-adjusts when determining how much to pay private Medicare Advantage insurance plans. It also used risk adjustments when deciding that 2,217 hospitals should be penalized for having high rates of patient readmissions.  Risk adjustment is also a key component in new models of delivering care, such as the accountable care organizations.

The new study by the Dartmouth Atlas Project, published today in the health journal BMJ, faults the practice of trying to assess how sick patients are by looking at records to see patient diagnoses. The authors argue that the more times patients see doctors or get tests, the more new diagnoses they are given. “The more one looks, the more one finds,” the authors wrote. The Atlas researchers have asserted in three decades of research that areas of the country with gluts of hospital beds, specialists and other providers tend to deliver more care, whether it’s needed or not.

“You would think sicker places would have higher visit rates, but they don’t,” said Dr. John Wennberg, the lead author and the founder of the Atlas.

Here’s how their latest study worked: The researchers examined Medicare records for more than 5 million beneficiaries in 306 different regions of the country. They looked at three different formulas commonly used to assess how sick patients are, each based on the number and nature of diagnoses for patients as well their age, race and sex. Medicare uses one of those methods, known as “hierarchical condition categories” (HCC) to adjust for risk.

The researchers also analyzed the death rates of patient populations in each of the 306 regions. They found that the sickness of the patients explained between 10 and 12 percent of the discrepancy between places with high mortality rates and those with low mortality rates. But there was still a wide spread between regions of the country. For instance, under the HCC method, the death rate in the Salt Lake City region was 59.3 patients per 1,000—much higher than around Miami, where the death rate was 32.6 patients per 1,000. If that difference were accurate, then it would appear that patients in Salt Lake City were getting astoundingly worse care than in Miami—something that the researchers considered implausible.

Next, the researchers looked at the number of physician visits the patients had in the previous year. They then used statistical methods to “correct” the sickness rates, essentially reclassifying those patients with lots of excess physician visits as less sick than they would appear based by their diagnoses alone.

When the researchers used this revised metric to look at regional death rates, they now found it explained between 21 percent and 24 percent of the differences between high-mortality and low-mortality areas—twice as much as the standard risk-adjustment methods explained. Once visits were factored into the equation, Salt Lake City’s death rate dropped to 51.8 patients per 1,000 and Miami’s rate rose to 47.3 percent. That was much closer than before, although there remained an unexplained variation.

In a phone interview, Wennberg said the paper showed that the government and others need to refine the methods of adjusting for risk. “The way we’re doing it now has a lot of problems,” he said.


Private insurers pride themselves on market innovation. They will always find ways to reduce the amount that they spend on patients. They use devious methods to selectively enroll healthier individuals while receiving payments that are more appropriate for a mixture of both the sick and the healthy. When efforts are made by means of risk adjustment to modify payments to compensate for this injustice, insurers will use data manipulations to make their patients appear to be even sicker than they are in order to receive extra payments for their care.

This study by John Wennberg and his colleagues demonstrates that the differing regional rates of visits by physicians introduces a bias that makes it appear that regions with lower visits by physicians have higher costs and higher mortality rates, and vice versa. They conclude that correcting for such variations in intensity of patient observation (physician visit rates) would improve current risk adjustment methodologies, but that this would still account for “less than 25% of geographic variation in age, sex, and race adjusted mortality among fee for service Medicare beneficiaries.”

We already know that the private Medicare Advantage plans play games with risk adjustment. The Affordable Care Act will require risk adjustment between the private plans offered by the state insurance exchanges, and we can anticipate that they, too, will game the system.

Will this latest study finally bring us a risk adjustment process that the insurers cannot game? Unlikely. As more data are added, such as the intensity of patient observation suggested by this study, the administrative complexity increases, while the insurers find ever more not-yet-patched holes in the risk adjustment infrastructure.

As long as individual patients are linked to individual private plans, there will always be intermediaries – the private insurers – who will manipulate the system to their own benefit. We should remove these superfluous, administratively inefficient middlemen and replace them with our own public administrators. The task of negotiating appropriate payments with health care professionals and institutions would be much simpler if we got the private intermediaries out of the way.