Decision Theoretic Approaches to Experiment Design and External Validity (Working Paper 22167)

By Abhijit Banerjee, Sylvain Chassang, and Erik Snowberg
National Bureau of Economic Research, April 2016

Abstract

A modern, decision-theoretic framework can help clarify important practical questions of experimental design. Building on our recent work, this chapter begins by summarizing our framework for understanding the goals of experimenters, and applying this to re-randomization. We then use this framework to shed light on questions related to experimental registries, pre-analysis plans, and most importantly, external validity. Our framework implies that even when large samples can be collected, external decision-making remains inherently subjective. We embrace this conclusion, and argue that in order to improve external validity, experimental research needs to create a space for structured speculation.

Formats for structured speculation

In all external decision making problems, inference is unavoidably subjective. In structural modeling, the source of subjectivity is the model itself.

http://www.nber.org/papers/w22167

***

Comment:

By Don McCanne, M.D.

This highly technical paper is a difficult read, but it provides important lessons for design and application of health policy research. The subjective design of the research model and subjectivity of the resulting decision making impacts the external validity of the experiment. That is, can the results of the experiment be applied generally outside of the experimental model?

Perhaps the most famous example is the RAND Health Insurance Experiment. It demonstrated that employees and their families who had to pay part of the costs of their health care used less health care than did those who had no out-of-pocket spending, yet both groups had equally good health care outcomes (except for hypertensives). This conclusion was internally valid for this group, and it has been applied to the population at large. But is it really externally valid?

Employees and their families are a comparatively healthy sector of society, and in the RAND HIE they were studied during a healthy period of their lives. These results would not apply to the population at large that includes less healthy individuals who help run up our three trillion dollar national health care bill. Yet we have accepted the results of the RAND HIE as gospel even though other studies have confirmed that cost sharing does cause individuals to forgo beneficial health care services.

In the Oregon Experiment individuals with no insurance were compared with individuals who received Medicaid benefits by lottery. The conclusions published in the NEJM were that “Medicaid coverage generated no significant improvements in measured physical health outcomes in the first 2 years,” and this has been interpreted widely to mean that Medicaid does not improve health. The internal validity was regarding three measurements – BP, cholesterol and HbA1c during only two years. How could that possibly have external validity for the low-income population at large with innumerable infirmities that we know respond to beneficial health care services?

The subjectivity of sectors of the policy community is at question here. Model design is subjective and inference is subjective. Studies can be designed and interpreted to obtain the results desired. Cost sharing can be shown to reduce spending without having any adverse effect on health. Medicaid can be shown to have no benefit on health outcomes.

And what do we have now? Apparently policy science is so advanced that we are putting in place measures that do not require research to confirm their effectiveness. Narrow provider networks can reduce spending by deterring patients who, after all, really do not need care, they say. High deductibles can reduce spending without adversely affecting health, except for those who deferred the health care that they should have had. They say accountable care organizations will improve quality and significantly reduce spending, even though experience to date has largely failed to confirm that. The Merit-Based Incentive Payment System (MIPS) will convert the volume of health care delivery to the elusive concept of value, they hope, though the limited evidence indicates otherwise.

What is the purpose of health policy research? To improve quality? Isn’t quality determined by studying the diagnostic, therapeutic and preventive interventions in heath care – i.e, medical research? Is the purpose of policy research to control spending? Except by creating detrimental financial and logistical (network) barriers to beneficial care, policy studies have been remarkably ineffective. Public policies such as establishing a single payer system have been far more effective in controlling spending.

The authors of this NBER paper state, “in order to improve external validity, experimental research needs to create a space for structured speculation.” Haven’t we had enough of this structured speculation of the health policy community? Isn’t it time to adopt public policies that have been proven to be effective?

Decision Theoretic Approaches to Experiment Design and External Validity (Working Paper 22167)

By Abhijit Banerjee, Sylvain Chassang, and Erik Snowberg
National Bureau of Economic Research, April 2016

Abstract

A modern, decision-theoretic framework can help clarify important practical questions of experimental design. Building on our recent work, this chapter begins by summarizing our framework for understanding the goals of experimenters, and applying this to re-randomization. We then use this framework to shed light on questions related to experimental registries, pre-analysis plans, and most importantly, external validity. Our framework implies that even when large samples can be collected, external decision-making remains inherently subjective. We embrace this conclusion, and argue that in order to improve external validity, experimental research needs to create a space for structured speculation.

Formats for structured speculation

In all external decision making problems, inference is unavoidably subjective. In structural modeling, the source of subjectivity is the model itself.

http://www.nber.org/papers/w22167

This highly technical paper is a difficult read, but it provides important lessons for design and application of health policy research. The subjective design of the research model and subjectivity of the resulting decision making impacts the external validity of the experiment. That is, can the results of the experiment be applied generally outside of the experimental model?

Perhaps the most famous example is the RAND Health Insurance Experiment. It demonstrated that employees and their families who had to pay part of the costs of their health care used less health care than did those who had no out-of-pocket spending, yet both groups had equally good health care outcomes (except for hypertensives). This conclusion was internally valid for this group, and it has been applied to the population at large. But is it really externally valid?

Employees and their families are a comparatively healthy sector of society, and in the RAND HIE they were studied during a healthy period of their lives. These results would not apply to the population at large that includes less healthy individuals who help run up our three trillion dollar national health care bill. Yet we have accepted the results of the RAND HIE as gospel even though other studies have confirmed that cost sharing does cause individuals to forgo beneficial health care services.

In the Oregon Experiment individuals with no insurance were compared with individuals who received Medicaid benefits by lottery. The conclusions published in the NEJM were that “Medicaid coverage generated no significant improvements in measured physical health outcomes in the first 2 years,” and this has been interpreted widely to mean that Medicaid does not improve health. The internal validity was regarding three measurements – BP, cholesterol and HbA1c during only two years. How could that possibly have external validity for the low-income population at large with innumerable infirmities that we know respond to beneficial health care services?

The subjectivity of sectors of the policy community is at question here. Model design is subjective and inference is subjective. Studies can be designed and interpreted to obtain the results desired. Cost sharing can be shown to reduce spending without having any adverse effect on health. Medicaid can be shown to have no benefit on health outcomes.

And what do we have now? Apparently policy science is so advanced that we are putting in place measures that do not require research to confirm their effectiveness. Narrow provider networks can reduce spending by deterring patients who, after all, really do not need care, they say. High deductibles can reduce spending without adversely affecting health, except for those who deferred the health care that they should have had. They say accountable care organizations will improve quality and significantly reduce spending, even though experience to date has largely failed to confirm that. The Merit-Based Incentive Payment System (MIPS) will convert the volume of health care delivery to the elusive concept of value, they hope, though the limited evidence indicates otherwise.

What is the purpose of health policy research? To improve quality? Isn’t quality determined by studying the diagnostic, therapeutic and preventive interventions in heath care – i.e, medical research? Is the purpose of policy research to control spending? Except by creating detrimental financial and logistical (network) barriers to beneficial care, policy studies have been remarkably ineffective. Public policies such as establishing a single payer system have been far more effective in controlling spending.

The authors of this NBER paper state, “in order to improve external validity, experimental research needs to create a space for structured speculation.” Haven’t we had enough of this structured speculation of the health policy community? Isn’t it time to adopt public policies that have been proven to be effective?