First Look At Medicare Quality Incentive Program Finds Little Benefit

By Jordan Rau
Kaiser Health News, August 6, 2014

One of Medicare’s attempts to improve medical quality – by rewarding or penalizing hospitals – did not lead to improvements in the first nine months of the program, a study has found.

The quality program, known as Hospital Value-Based Purchasing, is a pillar of the federal health law’s campaign to use the government’s financial muscle to improve patient care. Since late 2012, Medicare has been giving small increases or decreases in payments to nearly 3,000 hospitals based on how patients rated their experiences and how faithfully hospitals followed a dozen basic standards of care, such as taking blood cultures of pneumonia patients before administering antibiotics. …

The study, published last month on the Health Services Research journal online site, is the first to look at how hospitals performed under the value-based purchasing program. The researchers, led by Andrew Ryan, … analyzed hospitals’ performances in the five years before the program began and the period from July 2011 through March 2012, the nine months of data that Medicare used to determine the first year of bonuses and penalties. The researchers compared how the hospitals in the program did with the performance of several hundred hospitals that were exempted from the program. … The researchers found no significant difference in performance, with both groups of hospitals improving at equal rates.

Evidence refuting the claims made for pay-for-performance (P4P) is piling up. The paper by Andrew Ryan et al. is just the latest example. Ryan et al. found that yet another P4P experiment with hospitals, this one ordered by Congress in 2010, isn’t working. We already knew that. An enormous P4P experiment involving hospitals, ordered by Congress in 2003, didn’t work either. That project, known as the Premier Hospital Quality Incentive Demonstration, has been shown to have no effect on quality.

The best that can be said for P4P is that it can sometimes cause doctors to score slightly higher on measures linked to financial incentives, but only at the expense of patients whose care is not being measured. We know this teaching-to-the-test effect is real because research indicates doctors must work 22 hours a day to meet existing guidelines for preventive and chronic care and still address the acute care needs of their patients. Obviously, under those constraints, P4P can work only by shifting physician priorities away from priorities previously established by doctors and their patients.

The teaching-to-the-test effect of P4P was demonstrated at Kaiser Permanente Northern California which began experimenting with P4P in the late 1990s. Lester et al. found that removing incentives for screening for diabetic retinopathy and cervical cancer led to “a decrease in performance of about 3 percent per year on average for screening for diabetic retinopathy and about 1.6 percent per year for cervical cancer screening.”

Lester et al. speculated that they had demonstrated the teaching-to-the-test effect. In my view, they demonstrated it. Even assuming the worst about Kaiser Permanente’s doctors – that large numbers of them had no idea they should examine the retinas of diabetics or screen for cancer in women, and that this widespread ignorance was corrected by Kaiser’s P4P scheme – we must still conclude that the decline in the scores on these measures after the P4P incentives were removed reflected a return to physician-patient priorities uncontaminated by P4P incentives. The alternative explanation – that a substantial number of doctors reacquired a previous ignorance about the measured services, and there were no competing demands on their time – makes no sense.

Before wrapping up this indictment of P4P I should note that P4P harms poor people, and as of this date we have virtually no information on what it costs to administer the myriad P4P schemes in operation today.

It is timely, therefore, to ask, What happens when health policy experts commit errors? Do we say, “To err is human,” and move on?

Errors by doctors and hospitals trigger a variety of responses, including investigations, malpractice suits, loss of patients, loss of licenses, and criminal prosecution. But when health policy experts make mistakes, nothing happens, at least nothing potent enough to alter expert behavior. The P4P fad illustrates this double standard. P4P was endorsed by the Institute of Medicine (IOM) and many other prominent groups and individuals with no evidence to support it, and now that the evidence indicates P4P doesn’t work, not one of these groups and individuals has stepped forward to admit error.

We need an analysis of errors in health policy and why those errors go uncorrected long after they have been revealed. I would like to suggest that the IOM undertake this task. I suggest they entitle their report, To Err is Human, and Health Policy is No Exception.

In 1999, the IOM released a highly influential report entitled To Err is Human which estimated errors in hospitals caused 44,000 to 98,000 deaths per year.  The IOM said this was “not acceptable” and called for a “comprehensive strategy [to] reduce preventable medical errors.” It defined “medical errors” as “the failure of a planned action to be completed as intended or the use of a wrong plan to achieve an aim.”

The experts’ endorsement of P4P meets the IOM’s definition of “error”: “The use of a wrong plan to achieve an aim.” For several reasons, the P4P fad constitutes an especially good case study for the study of errors by health policy experts:

•  The fad had a rather discrete beginning date (about 2000),
•  it was clear when the fad began that there was no evidence to support it,
•  the proponents of the fad left behind a rather extensive paper trail revealing their flimsy justification for recommending it, and
•  evidence indicting the fad accumulated quickly.

The P4P fad sprang up around 2000 not because research suddenly demonstrated it would work, but because the insurance industry, self-insured employers, and their allies in government and academia wanted desperately to find a new cost-control tactic to replace the more intrusive tactics that inspired the “HMO backlash” of the 1990s. (See, for example, this comment by Paul R. Reich, MD, the medical director of Blue Cross and Blue Shield of Rhode Island, in a 2003 article: “With the decline of capitation as a means of compensating doctors, ‘paying for performance’ has become a viable alternative.” [“Pay for performance,” Managed Care Interface 2003;16:14])

By 2005 the health policy establishment’s interest in P4P had exploded into a full-fledged fad. The Leapfrog Group (a creation of the Business Roundtable) endorsed P4P in 2000, the IOM endorsed it in 2001, an association of eight insurers in California (the Integrated Healthcare Association) established a large P4P program in 2001, and Medpac endorsed P4P in reports to Congress published in 2003 and 2005. In 2003 a group of prominent managed care advocates, including Donald Berwick, Nancy-Ann DeParle, Paul Ellwood, Alain Enthoven, and John Wennberg, published a paper in Health Affairs entitled “Paying for Performance: Medicare should lead.” By 2005, according to the Congressional Research Service, at least 107 P4P programs were underway in the US in the private and public sectors, and Congress was considering authorizing more.

This call for P4P from the health policy elite was not supported by evidence that P4P in medicine was safe, effective, or affordable. For example, in the paper mentioned above by Berwick et al., the authors cited not one study supporting their assertion that “payment for performance should become a top national priority.”

P4P proponents did not justify their call for P4P with evidence because there was none to invoke. Rather, their justification boiled down to, “The status quo is terrible; P4P can’t be worse than the status quo.” Glenn Hackbarth, chairman of Medpac, offered that rationale for Medpac’s endorsement of P4P in a 2006 paper:

Why is MedPAC confident that P4P is the proper thing to do, especially given the limited amount of hard evidence on its impact? Two reasons. First, there is overwhelming research documenting the poor performance of our health care system…. The status quo is unacceptable…. Second, there is abundant evidence that health care providers respond to incentives. For people with substantial experience in health care delivery and policy, like the MedPAC commissioners, it does not seem like much of a leap to conclude that P4P is a step in the right direction.” (“Commentary,” Medicare Care Research and Review 63:1 (Supplement to February 2006), 118S.

In a 2005 press release, Robert Wood Johnson’s president and CEO offered a similar justification: “Whether or not you believe P4P will make a significant difference, it’s time for payers of health care to reward providers who are improving quality rather than turn a blind eye.” (“Pay for performance improving health care quality and changing provider behavior; but challenges persist,” press release, November 15, 2005.)

I will leave aside for now the question of whether P4P proponents have vastly exaggerated the “quality” problem and ignored the role that administrate waste (including the cost of running P4P projects), high prices, and managed care play in preventing patients from getting recommended medical care. I’ll note merely that if doctors adopted the same excuse for trying out risky, expensive and unproven treatments on their patients – “My patient was very sick, the status quo was unacceptable, I couldn’t turn a blind eye” – the health policy elite would raise holy hell, and rightly so. But the standards for “people with substantial experience in health care delivery and policy,” to quote Hackbarth, are different. If you are one of those people, raw, fervently held opinion will suffice.

This devil-may-care attitude toward evidence is a root cause of error in health policy. We badly need research on why this attitude exists and its role in our nation’s inability to adopt effective health policies.

Kip Sullivan, J.D., is a member of the steering committee of the Minnesota chapter of Physicians for a National Health Program. His writing has appeared in The New York Times, The Nation, The New England Journal of Medicine, Health Affairs, the Journal of Health Politics, Policy and Law, and the Los Angeles Times.