Project Evaluation Activity in Support of Partnership for Patients: Task 2 Evaluation Progress Report
Center for Medicare and Medicaid Innovation (CMMI)Submitted 7/10/2014
The Partnership for Patients (PfP) campaign was launched in April 2011 with the ambitious goals of reducing preventable hospital-acquired conditions (HACs) by 40 percent and 30-day hospital readmissions by 20 percent. To reduce harm at this level of magnitude, the campaign implemented a strategy to align all health care stakeholders, including federal and other public and private health care payors, providers, and patients, to focus on this issue concurrently. By influencing everyone to move in the same direction at the same time, the program strove to overcome the inherently limited reach of any single initiative operating in a complex environment. The three major components of the campaign, conceptualized as “engines,” are the Centers for Medicare & Medicaid Innovation (CMMI) investment engine, the federal partner alignment engine, and the outside partner engine. The program is national in scope, due to its level of implementation. For example, over 70 percent of general acute care hospitals in the United States (U.S.), representing over 80 percent of admissions, worked with PfP-funded Hospital Engagement Networks (HENs) during 2012-2013.
The PfP campaign focuses on 11 areas of patient harm. To date, the evaluation has found clear evidence for decreased rates of harms in five of the eleven areas, meaning the decreases are statistically significant, and/or meet statistical process control criteria for a special cause decrease, and/or (in cases where only aggregated data are available) are large in magnitude. These areas include obstetrical early elective deliveries (OB-EED), readmissions, adverse drug events (ADE), ventilator-associated pneumonia (VAP), and central line-associated bloodstream infection (CLABSI). In the other six areas, to date, the evaluation has found mixed evidence, meaning some datasets show decreases, while others show no change, or even worsening, including venous thromboembolism (VTE), catheter-associated urinary tract infection (CAUTI), other OB adverse events (OB-Other), pressure ulcers, surgical site infections (SSI), and falls.
The cost estimates available to date suggest cumulative savings of between $3.1 to $4 billion as a result of the decreases in harms since the baseline of 2010. Additionally, AHRQ has estimated 15,5001 deaths averted since 2010, based on mortality rate estimates associated with targeted harms. Tables 1 and 2 synthesize the evidence available to date for improvement in the rate of adverse events in each of the 11 areas, and Table 3 provides cost reduction estimates from the two available sources of estimates to date. Since hospital payment policies and other U.S. Department of Health & Human Services (HHS) programs that played an important role as part of the PfP campaign were in place and making changes over time, it is not possible at this time for the evaluation to identify the portion of these harm reductions and savings attributable to the PfP campaign’s direct work with hospitals versus alignment of forces for harm reduction versus other harm reduction work that would have continued with or without PfP.
About the CMS Innovation Center
The Innovation Center was established by section 1115A of the Social Security Act (as added by section 3021 of the Affordable Care Act). Congress created the Innovation Center for the purpose of testing “innovative payment and service delivery models to reduce program expenditures …while preserving or enhancing the quality of care” for those individuals who receive Medicare, Medicaid, or Children’s Health Insurance Program (CHIP) benefits.
Did Hospital Engagement Networks Actually Improve Care?
By Peter Pronovost, M.D., Ph.D., and Ashish K. Jha, M.D., M.P.H.
The New England Journal of Medicine, August 21, 2014
Everyone with a role in health care wants to improve the quality and safety of our delivery system. Recently, the Centers for Medicare and Medicaid Services (CMS) released results of its Partnership for Patients Program (PPP) and celebrated large improvements in patient outcomes. But the PPP’s weak study design and methods, combined with a lack of transparency and rigor in evaluation, make it difficult to determine whether the program improved care. Such deficiencies result in a failure to learn from improvement efforts and stifle progress toward a safer, more effective health care system.
CMS launched the PPP in December 2011 as a collaborative comprising 26 “hospital engagement networks” (HENs) representing more than 3700 hospitals, in an effort to reduce the rates of 10 types of harms and readmissions. The HENs work to identify and disseminate effective quality-improvement and patient-safety initiatives by developing learning collaboratives for their member facilities, and they direct training programs to teach hospitals how to improve patient safety. In a February 2013 webcast, CMS announced that the rates of early elective deliveries had dropped 48% among 681 hospitals in 20 HENs and that the national rate of all-cause readmissions had decreased from 19% to 17.8%, though it is unclear which HENs were included for each measure and what time periods were the pre- and post-intervention periods.
These numbers appear impressive, but given the publicly available data and the approach CMS used, it’s nearly impossible to tell whether the PPP actually led to better care. Three problems with the agency’s evaluation and reporting of results raise concerns about the validity of its inferences: a weak design, a lack of valid metrics, and a lack of external peer review for its evaluation. Though the evaluation of many other CMS programs also lacks this basic level of rigor, given the large public investment in the PPP, estimated at $1 billion, and the strong public inferences about its impact, the lack of valid information about its effects is particularly troubling.
The design of a quality-improvement program influences our ability to make reasonable inferences about its benefits to patients. Although individual HENs may have used more rigorous methods, the overall PPP evaluation had three important weaknesses: it used a pre–post design with only single points in the pre and post periods, did not have concurrent controls, and did not specify the pre and post periods a priori. Such an approach is highly subject to bias.
There are alternatives available, including a randomized or even a cluster-randomized trial. If such trials were not feasible, CMS could have used other robust design approaches, such as an interrupted time-series study with concurrent controls. Rather than having a single pre time period and a single post time period, this design entails repeated measurements of the safety indicators before and after the intervention in both HEN and non-HEN hospitals. Such an approach would have provided more valid inferences about the effects of the program, with few additional costs.
Beyond using a poor design, CMS did not use standardized and validated performance measures across all participating hospitals — further hampering inferences about the program’s effects. To support engagement, CMS allowed each HEN to define its own performance measures, with little focus on data quality control.
CMS also required HENs and participating hospitals to submit a large number of process measures of unknown validity. It is essential to use validated measures — ideally those endorsed by the National Quality Forum — unless there is a compelling reason not to. In instances where validated measures are unavailable, instead of using poor quality metrics, CMS can have an agency such as the Agency for Healthcare Research and Quality (AHRQ) or the CDC develop measures rapidly.
Finally, CMS made — and presented publicly — inferences about its program’s benefits without having subjected its work to independent evaluation or peer review. Peer review, though imperfect, is a powerful quality control.
The PPP involved an investment of nearly $1 billion to improve care — three times the annual budget of the AHRQ, the lead federal funding agency for implementation science, which often lacks resources for promising projects. With such a sizable investment, CMS could have supported a better evaluation. It could have randomized HENs or hospitals to receive interventions earlier or later; used standardized, validated measures across the HENs; built in basic data quality controls; and independently collected qualitative information alongside quantitative data to learn not just whether the interventions worked but also how and why they did, thereby advancing our understanding of the mechanisms and context of improvement science. These changes would have allowed the country to learn so much more.
The lack of a careful evaluation is symptomatic of a broader problem: some members of the quality-improvement community eschew even modestly rigorous methods, believing that one can simply “know” if an intervention worked. Though maintaining hope and optimism among clinicians is important, when untested interventions are implemented widely, they often fail to improve care. The confidence we can have in an intervention’s efficacy is directly related to the rigor with which it is designed, implemented, and evaluated. Given the strong desire to improve care and the conflicts of interest we all face in evaluating our own work, subjecting all evaluations to external examination is critical.
The field of improvement science is still in its infancy. Given the magnitude of the quality and cost problems in health care and the amount of money invested in mitigating these problems, the public, providers, and policymakers need to have confidence that money used to improve care is being well spent. It’s true that improvement science requires mixed methods and is difficult, but all good science is difficult. Failing to attend closely to issues of design, methods, and metrics leaves us with little confidence in an intervention. For the PPP, which required thousands of hours of clinicians’ time and large sums of money, that lack of confidence is particularly unfortunate. More important, the failure to generate valid, reliable information hampers our ability to improve future interventions, because we are no closer to understanding how to improve care than we were before the PPP. And that is the biggest cost of all.
By Don McCanne, MD
Another creation of the Affordable Care Act (ACA) is the Center for Medicare and Medicaid Innovation (CMMI) – an entity established to test innovations in payment and service delivery models designed to reduce costs and improve quality. How is it doing?
After spending almost a billion dollars on a study designed to reduce hospital-acquired conditions – a budget three times the total annual budget of AHRQ (Agency for Healthcare Research and Quality) – we have almost nothing to show for that effort and expense. As the CMMI report states, “Since hospital payment policies and other U.S. Department of Health & Human Services (HHS) programs that played an important role as part of the PfP campaign were in place and making changes over time, it is not possible at this time for the evaluation to identify the portion of these harm reductions and savings attributable to the PfP campaign’s direct work with hospitals versus alignment of forces for harm reduction versus other harm reduction work that would have continued with or without PfP.”
In their article on the flaws in this program, Peter Pronovost and Ashish Jha make an observation that typifies what has been wrong with the entire reform process centered on ACA. They state, “some members of the quality-improvement community eschew even modestly rigorous methods, believing that one can simply “know” if an intervention worked. Though maintaining hope and optimism among clinicians is important, when untested interventions are implemented widely, they often fail to improve care.”
Think of some of the prominent personalities involved in crafting and implementing ACA and how outspoken they were and continue to be on what they simply “know” will work – accountable care organizations, bundled payments, pay for performance, competing exchange plans bringing us higher quality at lower cost, placing the empowered consumer in charge through deductibles and other cost sensitivity, and improving payment policies through the Center for Medicare and Medicaid Innovation.
The tragedy is that much of this was to avoid adopting a program that every informed person knows really would work – an improved Medicare for all. It would have been far better to have directed that billion dollars towards implementing single payer.