Putting economic evaluation into practice

Economic evaluation is a useful tool to help evaluate social interventions. The goal is to put dollar values on the impacts of the intervention and compare these to its costs. This is known as “social cost-benefit analysis” (SCBA), “social return on investment” (SROI) or just economic evaluation. The basic principles come from welfare economics. Benefits counted in an economic evaluation can include improvements in human health and wellbeing, and the natural environment. (Economics is not about money! It’s about how we make choices to use our resources to get things that we need and want.)

If done well, an economic evaluation tests if the value of the outcomes and impacts that an intervention created are greater than its costs, and if so by how much. The end result can be expressed as a “benefit-cost ratio” (BCR) which is just total benefits divided by costs. The BCR says how much benefit was created for each dollar of cost. This is sometimes called the social return on investment: “for each $1 of costs, the intervention created $X of social benefits”.

Economic evaluation is data-intensive because it must overcome two key challenges:

  1. Causality: The changes in outcomes caused by the intervention must be estimated. This involves comparing outcomes with the intervention against outcomes under a suitable counterfactual. But counterfactual outcomes can’t be observed directly, so they need to be estimated. Suitable data and statistical methods need to be used to estimate counterfactual outcomes and minimise bias due to factors aside from the intervention that may also have affected outcomes. This is a challenge for any quantitative evaluation of outcomes, not just economic evaluation.
  2. Valuation: Dollar values need to be assigned to changes in outcomes caused by the intervention (typically these are its benefits, but can also be costs). These values need to reflect society’s “willingness to pay” for the relevant changes in outcomes. Most types of social outcomes are not bought and sold in markets, so we don’t have direct information about willingness to pay. Empirical economic methods need to be used to estimate valuation factors that apply to the types of changes in outcomes caused by the intervention.

Doing economic evaluation well requires good data and information, and robust methods. The biggest problem that I’ve encountered in practice is a lack of suitable data. Often this is because data requirements for economic evaluation were not considered until an evaluation is been commissioned after an intervention has ended. By that stage it can be difficult to gather the data needed. It’s much better to start thinking about this before the intervention is even implemented.

For example, one reasonably robust method for addressing causality is to compare changes in outcomes before and after the intervention across a group supported by the intervention and a matched comparison group. This requires data on outcomes for both groups at two points in time: before and after the intervention. If you only consider data requirements after the intervention is finished, you won’t be able to collect the data on pre-intervention outcomes.

As I said, it’s critical that the data required for economic evaluation is considered at an early stage. To help with this, I’ve written a brief guide to the data and information requirements for economic evaluation of social interventions. I hope it will be a useful resource for those hoping to plan and commission economic evaluations. Even if you don’t do the valuation part, most of the data requirements also apply to quantitative evaluation where you want to estimate causal impacts of an intervention.

These issues sit at the intersection of data science and economics, which is where I work. Please get in touch if you need help.