Позитивные изменения. Том 2, № 2 (2022). Positive changes. Volume 2, Issue 2 (2022)

The challenge is to find a method that will allow us to establish a causal relationship. We must empirically determine to what extent a particular program — and that program alone — contributed to the change in outcome. The methodology must exclude all external factors.

The answer to the basic question of impact evaluation is what is the impact or causal effect of the program (P) on the outcome of interest (Y)? — is given by the basic impact evaluation formula:

Δ = (Y | P = 1) − (Y | P = 0).

This formula states that the causal effect (Д) of the program (P) on the outcome (Y) is the difference between the outcome (Y) with the program (in other words, when P = 1) and the same outcome (Y) without the program (i.e. when P = 0).

For instance, should P denote a training program and Y denote income, then the causal effect of the training program (Д) is the difference between a trainee’s income (Y) after participating in the training program (in other words, when P = 1) and the same person at the same point in time if he or she did not participate in the program (in other words, when P = 0).

If this were possible, we would observe how much income the same person would have at the same point in time both with and without the program, so that the program would be the only possible explanation for any difference in that person’s income. By comparing the same person with himself or herself at the same time, we would be able to exclude any external factors that could also explain the difference in outcomes.

But unfortunately, measuring two versions of the same unit at the same time is impossible: at a particular point in time, the person either participated or did not participate in the program.

This phenomenon is called the counterfactual problem: how can we measure what would happen if other circumstances prevailed?

COMPARISON AND TREATMENT GROUPS

In practice, the task of impact evaluation is to identify a comparison group and a treatment group that are similar in their parameters, but one of them participates in the program and the other does not. That way any difference in results must be due to the program.

The treatment and comparison groups should be the same in at least three respects:

1. The baseline properties of the groups should be identical. For example, the mean age of the treatment group should be the same as that of the comparison one.

2. The program factor should not affect the comparison group directly or indirectly.

3. The results in the comparison group should change in the same way as the results in the treatment group if both groups were (or were not) enrolled in the program. That is, groups should respond to the program in the same way. For example, if the income in the treatment group increased by RUB 5,000 due to the training program, then the income in the comparison group would also increase by RUB 5,000 if they received the training.

When the above three conditions are met, then only the existence of the program under study will account for any differences in the outcome (Y) between the two groups.

Instead of considering the impact solely for one person, it is more realistic to consider the average impact for a group of people (Figure 1).

It is important to consider what happens if we decide to proceed with the evaluation without finding a comparison group. We may run the risk of making inaccurate judgments about program outcomes, in particular with regard to counterfactual evaluations.

Such a risk exists when using the following approaches:

• Before-and-after comparisons (also known as reflexive comparisons): comparing the outcomes of the same group prior to and subsequent to the introduction of a program.

• With-and-without comparisons: comparing the outcomes in the group that chose to enroll with the results of the group that chose not to enroll.

A before-and-after comparison attempts to establish the impact of the program by tracking changes in outcomes for program participants over time. In essence, this comparison assumes that if the program had never existed, the outcome (Y) for program participants would have been exactly the same as their situation before the program. Unfortunately, in the vast majority of cases that assumption simply does not hold.

Consider, for example, the evaluation of a microfinance program for rural farmers. The program provides farmers with microloans to help them buy fertilizer to increase rice production. You observe that in the year before the start of the program, farmers harvested an average of 1,000 kilograms (kg) of rice per hectare. (Point B in Figure 2).