Позитивные изменения. Том 2, № 2 (2022). Positive changes. Volume 2, Issue 2 (2022)

Increasing incomes, improving the education system, reducing morbidity are important areas of impact investment. Whether these changes will actually be achieved is the key question for an investor deciding on a social technology or project to invest in. However, the leaders of social projects and programs often focus on measuring the immediate outputs rather than on assessing whether projects and programs have had the expected impact. In this article, we would like to highlight the experience of evaluating the impact of the Health Insurance Subsidy Program (HISP) and describe approaches that can be used to address this and other similar problems.

Elena Avramenko

Expert of the project "Development of a Social and Economic Impact Assessment Model for NGOs" by the GLADWAY Foundation, Lean 6 Sigma Green Belt master

HEALTH INSURANCE FOR THE POOR

The Health Insurance Subsidy Program (HISP) is a program implemented in Kenya that finances the purchase of health insurance for low-income households in rural areas. The insurance covers the costs associated with medical care and medications. The objective of HISP is to reduce out-of-pocket health expenditures of poor families and ultimately to improve health outcomes.

HISP was originally launched in pilot mode. The plans for gradual expansion of the program depended on the results of the pilot stage. As part of the pilot run, the plan was to reduce the average yearly per-capita health expenditures of poor rural households by at least USD 10 below what they would have spent in the absence of the program, and this target was to be reached within two years.

During the initial pilot phase, HISP was introduced in 100 rural areas. Of the 4,959 households in the baseline sample, a total of 2,907 were enrolled in HISP, and the program operated successfully through its pilot stage over the next two years. All health clinics and pharmacies serving 100 villages accepted patients under the insurance program, and surveys showed that most enrolled households were happy with the program. Data was collected before the start of the pilot run and at the end of the two-year period, using the same sample of 4,959 households.

PROOF OF IMPACT

Has HISP affected out-of-pocket health expenditures of poor rural households? Yes it has, and it has been proven mathematically. The impact evaluation approach used as part of HISP was to select the most rigorous method, given the specifics of the project.

HISP implementation case study provides us with the following «menu» of options for impact evaluation methods:

• randomized assignment;

• instrumental variables;

• regression discontinuity design;

• difference-in-differences;

• benchmarking method.

All of these approaches aim at identifying valid comparison groups so that the true impact of the program on out-of-pocket health care expenditures of poor households can be evaluated.

So, we build on the stage when the evaluation indicators are selected and elaborated in detail, the data collection plan is ready and the data is collected properly.

We will review the evaluation methodology selected for this case by introducing the concept of counterfactual (that is, a fact that contradicts the hypothesis). And then, within the framework of this article, we will give an overview of the most rigorous evaluation method proposed by HISP and tested on this program.

There are two concepts that are integral to the process of making accurate and reliable impact evaluations — the concept of causation and that of counterfactual.

First of all, issues of social impact are related to causation, for example, with the search for answers to such questions:

Does teacher training improve student test scores? Do additional funding programs for health facilities result in better health outcomes for children? Do vocational training programs increase a trainee's income?

Finding answers to these questions can be difficult. For example, in the context of a vocational training program, simply observing how a trainee’s income increases after completing such a program is not sufficient to establish a causal relationship. A trainee’s income could have increased even if he or she had not been trained — all through the trainee’s own efforts, due to changing conditions in the labor market, or due to many other factors that could affect income.