By Bernard Black, Alex Hollingsworth, Leticia Nunes, and Kosali Simon
National Bureau of Economic Research, February 2019
Abstract
A large literature examines the effect of health insurance on mortality. We contribute by emphasizing two challenges in using the Affordable Care Act (ACA)’s quasi-experimental variation to study mortality. The first is non-parallel pretreatment trends. Rising mortality in Medicaid non-expansion relative to expansion states prior to Medicaid expansion makes it difficult to estimate the effect of insurance using difference-in-differences (DD). We use various DD, triple difference, age-discontinuity and synthetic control approaches, but are unable to satisfactorily address this concern. Our estimates are not statistically significant, but are imprecise enough to be consistent with both no effect and a large effect of insurance on amenable mortality over the first three post-ACA years. Thus, our results should not be interpreted as evidence that health insurance has no effect on mortality for this age group, especially in light of the literature documenting greater health care use as a result of the ACA. Second, we provide a simulation-based power analysis, showing that even the nationwide natural experiment provided by the ACA is underpowered to detect plausibly sized mortality effects in available datasets, and discuss data needs for the literature to advance. Our simulated pseudo-shocks power analysis approach is broadly applicable to other natural-experiment studies.
From the Introduction
We do not find a statistically significant pattern of results consistent with Medicaid expansion causing mortality changes, but we also cannot rule out large effects in either direction. We note here that prior evidence on the effects of insurance expansion on mortality lead one to expect modest effects of incremental insurance expansions on mortality. Reasons for modest overall effects include: many of those in greatest need of healthcare are already insured; and many uninsured persons already receive substantial healthcare. One reason for our “null result” is that the first stage on insurance coverage is weak: our principal identifying variation (the relative change in uninsurance rates for Medicaid expansion versus non-expansion states) amounts to a very small fraction of the population. The average increase in health insurance coverage attributable to Medicaid expansion over 2014-2016 is only around 1.1% for persons aged 50-64, and only around 4% even when we hone in on low-educated populations; precise income measures used to determine ACA eligibility are unavailable in mortality data. A second reason for failure to reject the null of no effect is a high level of “noise” – substantial background variation in mortality, and mortality trends, across states and demographic groups. A third reason is that mortality is a low-frequency outcome. We note too that effects of health insurance on mortality are more likely to emerge over a long time frame.
Our second contribution is to highlight that observational studies can often benefit from performing and reporting power analyses. We use a simulation-based power analysis, in which we impose treatment effects of varying sizes on actual data during the pre-treatment period, and assess whether ACA expansion effects on mortality of plausible size can be detected with our data. We conclude that even the nationwide natural experiment provided by the ACA is severely underpowered to detect effects on mortality of plausible size in county-level death certificate data. To reliably detect effects of insurance coverage on mortality, one would need very large-sample panel data on individuals, which is not currently available. Such data could include information on health, income, and insurance status, which would allow the study to focus on subsamples with a larger first stage and/or higher sensitivity of health and mortality to healthcare use. Even with such hypothetical data, it is likely that only fairly large effects of health insurance on mortality could be reliably detected.
From the Discussion
In this paper, we examine the relationship between mortality and health insurance, principally using the DD research design used in many prior ACA studies. This design exploits the natural experiment created by variation between those states that expanded Medicaid insurance and those that did not. We also exploit variation that results from counties having varying uninsurance or poverty levels prior to 2014. We focus on persons aged 55-64 years, whose mortality rates are the most likely to be affected by health insurance. We study effects of the first three years after expansion by type of mortality (healthcare amenable vs non amenable), demographics (gender, race, and ethnicity), education level, cause of death, and residence in counties most likely to gain from the ACA expansion).
We find large confidence intervals with no statistically significant evidence of an ACA- induced decline in mortality in Medicaid expansion states. Instead, there are important non-parallel pre-treatment trends, and standard errors are far too large to allow detection of effects of plausible sizes. We confirm lack of power through a formal, simulation-based power analysis.
While it is possible that the mortality effect of the ACA health insurance expansion variation we study may materialize with more time, other factors make it unlikely they too could be statistically detected; lengthening the study period would increase likelihood that other sources of variation, including cross-border moves, the instability of insurance status over time, and the underlying causes of the non-parallel pre-treatment trends we observe, will pose challenges for credible causal inference. Moreover, our power analysis implies that an extra few years would still be insufficient to attain reasonable power, given plausible effect sizes.
Comment:
By Don McCanne, M.D.
This meticulous, technically complex, 187 page economic paper attempted to look at changes in amenable mortality (mortality that could be reduced by beneficial interventions) from implementation of the Affordable Care Act. Their somewhat shocking result: “Our estimates are not statistically significant, but are imprecise enough to be consistent with both no effect and a large effect of insurance on amenable mortality over the first three post-ACA years.”
Before discussing this further, it is important to take note of the authors’ caveat: “Thus, our results should not be interpreted as evidence that health insurance has no effect on mortality for this age group, especially in light of the literature documenting greater health care use as a result of the ACA,” and keep this in mind when others report that this study failed to show any benefit of ACA. Stating that the study showed no statistically significant benefit is not the same as stating that there was no benefit.
Could it be that health care provides no benefit? Clearly not. Management of major trauma, use of antibiotics for potentially lethal diseases, surgical interventions for potentially disabling of lethal disorders, preventive measures that reduce the risk of chronic diseases, and vaccines that prevent epidemics of serious diseases are just a few examples of the benefits of modern health care. Of course, health care is dynamic in that new beneficial measures are continually introduced and measures that prove to be detrimental are abandoned.
So can it be that health insurance provides no benefit? The answer here is a little bit more complicated. Health insurance is clearly beneficial when it prevents financial hardship for individuals who clearly need health care. Yet it can be detrimental when it erects financial barriers to care or when it impairs access by denying care outside of the insurer’s provider networks. Also wasting funds on administrative excesses can be detrimental when those funds could have been diverted to more beneficial purposes.
In the case of the implementation of ACA, some people did benefit by gaining insurance protection while avoiding financial hardship through government subsidies or through enrollment in Medicaid. On the other hand, some faced greater financial risk by being shifted to plans with lower actuarial value (paying a lower percentage on the medical bills), plus some lost access to their usual health care because of narrow provider networks. There is some suggestion that care was impaired for many Medicaid beneficiaries who were transferred to private Medicaid managed care plans that seemed to be more concerned about cost containment rather than care provision.
Could these deficient policies of ACA have resulted in the failure to come to statistically significant conclusions on the impact of ACA on amenable mortality? Although that is highly unlikely it would be interesting to compare the impact of a well designed single payer Medicare for All system on amenable mortality since such a system is crafted to benefit patients and not designed to enhance the market success of insurers and other sectors of the medical-industrial complex. But then, amenable mortality should not be the sole criterion for judging the beneficial effects a health care financing system. Good health outcomes and financial security also matter.
Stay informed! Visit www.pnhp.org/qotd to sign up for daily email updates.