Pioneer ACO Evaluation Findings from Performance Years One and Two

L&M Policy Research,  March 10, 2015

The providers affiliated with Pioneer ACOs may change between performance years. … Because beneficiaries are aligned to the ACO based on a plurality of E&M [evaluation and management] services from ACO-participating providers, changes in ACO provider composition can impact which beneficiaries are aligned to the ACO in a given year. … Table 23 shows the number of these providers affiliated with [23 Pioneer] ACOs in 2012; the number of providers lost through attrition after 2012; the number of newly affiliated providers in 2013; the percent of continuously affiliated providers from 2012 to 2013; and the percent of beneficiaries aligned with an ACO in 2013 who were also aligned in 2012. [pp. 93-94] … The proportion of 2013 participating providers who participated with the same ACO in 2012 ranged from 0.47 to 0.92, with an average of 0.73. [p. 96]

pioneer aco turnover table

The annual “churn” rate among Medicare accountable care organization (ACO) doctors and assigned patients is enormous: It averages around one-third for both doctors and patients. Because of this constant doctor and patient turnover, ACOs lose the majority of their assigned patients over a five-year period. How is an ACO supposed to be held “accountable” for services given to such a rapidly changing panel of patients by such a rapidly changing roster of physicians?

This question was totally ignored by ACO proponents before the concept was invented at a November 2006 meeting of the Medicare Payment Advisory Commission. It has been almost totally ignored since by ACO proponents and health policy researchers. We have only a few reliable reports on churn rates among Medicare recipients assigned to Medicare ACOs by CMS, and virtually no research on churn rates among those under 65 in commercial or Medicaid ACOs.

The two most credible reports on the problem appeared in evaluations of ACO Medicare programs for CMS: the evaluation of the first two years of the Pioneer ACO program by L&M Policy Research released by CMS last May, and the final report on the Physician Group Practice (PGP) Demonstration published by CMS in 2012. The PGP demo is widely regarded as a test of the ACO concept.[1]

L&M reported doctor and patient retention rates for the second performance year (2013) of the Pioneer ACO program. The average retention rate for doctors was 73 percent, and for patients 62 percent (see Table 23 presented above).[2]  In other words, of the doctors listed by the ACOs at the beginning of 2012 as participating ACO doctors, only 73 percent were listed as participating at the beginning of 2013. Similarly, of the Medicare recipients assigned by CMS to ACOs as of January 2012, only 62 percent remained assigned in January 2013. This means the churn rate (the opposite of the retention rate) was 27 percent for doctors and 38 percent for patients.

Data from the PGP demo suggests that the churn rate for Pioneer ACOs will not change substantially in future years. The patient attribution algorithm for the PGP demo was almost identical to the algorithm CMS is using for the Pioneer program: Medicare beneficiaries were assigned to PGPs based upon the plurality-of-primary-care-visits method – if the primary care doctor who saw the patient the most often during a baseline period was in a PGP, CMS assigned that patient to that PGP.[3]

The final evaluation of the PGP demo reported that annual patient churn rates for the 10 PGPs stayed at approximately 30 percent over a five-year period (see Table 11-2b here, p. 222). (The report did not reveal churn rates for doctors.) Here is how the report summarized the data: “PGPs generally retained approximately 70 percent of their assigned beneficiaries from one year to the next; and … PGPs generally retained approximately 40 percent of their assigned beneficiaries after five years.” (p. 221)

If an average annual patient churn rate of 30 percent results in a loss of 60 percent of the original patient pool over five years, a churn rate of 38 percent (the churn rate for the Pioneer ACOs over the first two years) should produce an even higher loss over five years.

For any ACO program in which patient attribution is determined by the plurality-of-primary-care-visits method, patient churn rates will be largely determined by patient “leakage” rates. Leakage rates are the rates at which patients in a given ACO visit primary care doctors outside the ACO during the previous year (or whatever period the administrator of the ACO has chosen to measure plurality of services).[4]  If the estimates of patient churn rates we have just reviewed – 38 percent for the Pioneer ACOs and 30 percent for the PGPs – are in the ballpark, we should expect to find patient leakage rates of roughly the same magnitude. In fact we do.

Valerie Lewis et al. reported a 31 percent patient leakage rate for simulated Medicare ACOs. For their simulation, Lewis et al. used the same assignment method CMS used in the Pioneer and PGP experiments – the plurality of E&M visits during a baseline period. Lewis et al. reported that 31 percent of the patients assigned to ACOs did not visit a primary care doctor within the ACO.[5]

What little data we have for the non-elderly population indicates the churn rate is higher for that population. A paper by HealthPartners, the third largest insurance company in Minnesota, analyzed 2010 claims data to determine what would happen if HealthPartners were to allocate all the patients seen by its 52 clinics to an ACO using several formulas, including the plurality-of-E&M-visits method. The authors reported an annual patient churn rate of 47 percent using the plurality method. This rate is much higher than the rates reported for the Pioneer program (38 percent), the ACOs simulated by Lewis et al. (31 percent), and the PGP demo (30 percent).[6]

In a 2011 article on the ACO fad, Kaiser Health News characterized the ACO as a notion that hadn’t been thought through. Paraphrasing a scholar at the Brookings Institution, the article stated, “The health industry tends to operate with a ‘kind of a herd behavior,’ rushing to implement an idea ‘without working through the detailed business questions of how they’ll work.’” The problem of high churn rates is the premier example of a fundamental issue “the herd” failed to think through before stampeding off to implement ACOs.

The supreme irony of high churn rates is that they make “accountability” impossible at all levels – the doctor, ACO, regulator, legislative, and policy entrepreneur levels. This is most obvious with respect to assigned patients ACO providers never see – let’s call them “phantom patients.” But it’s true as well for patients assigned to ACOs for only a short period of time – let’s call them the “short-term patients.” Finally, it’s even true to some degree for patients who churn in and out of the ACO over a period of years – let’s call these patients “revolving-door patients.’

If ACOs should not and cannot hold doctors accountable for patients they never or rarely see, and if a substantial portion of “attributed” patients in fact are phantom, short-term, or revolving-door patients, then ACOs cannot and should not be held accountable by CMS, members of Congress, and other policy makers for the cost and quality of care provided to their assigned patients. And if ACOs cannot be held accountable, then ACO proponents cannot be held accountable for their role in fooling “the herd” into rushing off to promote ACOs. ACO proponents can go on making their claims for ACOs year after year and no one can prove them right or wrong.

The unfairness and irrationality of attempting to hold ACOs accountable for their treatment of so many phantom, short-term and revolving-door patients is aggravated by the fact that CMS’s patient-attribution method appears to guarantee that ACOs have healthier patients to work with.[7]  Healthier patients tend to have more stable sources of care than sicker patients. It doesn’t matter which way the causality runs (from healthy patients to greater continuity of care, or from continuity of care to healthier patients). Whichever way it runs, the result of assigning to ACOs those patients who are more loyal to a particular clinic is to guarantee that ACOs get a healthier pool of patients to work with.

Those who argue that accurate risk adjustment can solve at least the biased selection problem are deluding themselves. The best risk-adjustment methods available today predict very little of the variation in cost and quality outcomes among patients.

Sooner or later ACO proponents must recognize that high churn rates make ACO accountability impossible. When they do, they will have two choices. They can either call off the ACO experiment, or they can change the definition of ACO to include a requirement that patients enroll in an ACO for at least a one-year period and suffer penalties if they seek care outside the ACO’s network. If ACO proponents make the latter choice, they will in effect admit that what they have wanted all along was a re-run of the failed HMO experiment.


1. Even ACO proponents viewed the PGP demo that way. Like the Pioneer ACO program, the groups selected by CMS to participate in the PGP demo were large organizations that had years of experience using managed care tactics. Seven of the 10 groups owned or previously owned an HMO.

2. These churn rates were for the 23 ACOs that were still in the program as of the end of 2013. Nine ACOs, out of 32 that began the program in January 2012, dropped out during 2013. (A total of 13 have dropped out to date.) L&M calculated the average retention rate for doctors based on the data for each of the 23 ACOs reported in Table 23. I calculated the average for patients using the same method.

3. The PGP demo used what is called “retrospective” assignment while the Pioneer ACO program uses “prospective” assignment. For purposes of determining churn rates, this distinction does not matter. What matters is that both experiments assigned patients based on the plurality-of-E&M-visits method.

4. Consultants who advertise their ability to reduce “leakage” are usually referring to the rates at which patients in a given ACO visit any doctor or hospital outside the ACO In this comment, I define leakage more precisely as the rate at which patients visit primary care doctors outside their ACO.

5. Lewis et al. did not use the word “leakage” to describe this phenomenon, and they did not indicate whether the entire 31 percent who failed to seek care from an ACO-affiliated primary care doctor visited doctors outside the ACO. But given that only 6 percent of those 65 and older visit no doctor in the course of a year (see Table 78 here), we may deduce that a large majority of that 31 percent sought care from primary care doctors outside their ACO.

6. The higher rate is probably due to the fact that the non-elderly see primary care doctors less often than the elderly do, and a large portion of the non-elderly visit no doctors at all in any given year. According to the 2011 report to the Vermont legislature by William Hsiao et al. (in which the authors recommended a multiple-ACO system), “approximately 40 percent of covered individuals do not have any contact with a primary care physician in a one-year period.” (p. 159)

7. According to the final evaluation of the PGP demo, the plurality-of-E&M-visit method guaranteed the PGPs healthier populations in Year One of the demo. The average risk score for the Medicare recipients assigned to the PGPs that year was 0.921 (1.0 equaled average risk). Interestingly, the report demonstrated that these risk scores would have risen to approximately average if the method of assignment had been merely “one or more visits,” and would have fallen to 0.898 if “a majority of visits” had been used (see tables at page 222 of the final report). L&M Policy took note of this issue as well. They suggested, but did not demonstrate, that CMS’s assignment method may assign healthier patients to ACOs.

Kip Sullivan, J.D., is a member of the board of Minnesota Physicians for a National Health Program. His articles have appeared in The New York Times, The Nation, The New England Journal of Medicine, Health Affairs, the Journal of Health Politics, Policy and Law, and the Los Angeles Times.