- News
- Services
- Methods
-
Experience
- Cohesion Policy
- Labour market, Social Inclusion and Equal Opportunities
- Economy and Business
- R&D and Innovations
- Regional and Rural Development
- Communications and Information Society
- Science and Culture
- Improvement of Management and Administration
- Financial Instruments
- International Cooperation and Assistance
- Team
- Career
- Contact us
Methods
Counterfactual analysisCounterfactual analysis
In the counterfactual analysis, the outcomes of the intervention are compared with the outcomes that would have been achieved if the intervention had not been implemented. The method of counterfactual impact evaluation allows to identify which part of the observed actual improvement (e.g. increase in income) is attributable to the impact of the intervention (since such improvement might occur not only due to the intervention but also due to other factors, e.g. overall economic growth).
In order to identify the outcomes that would have been achieved in the absence of the intervention, treatment and control groups are formed. Treatment group consists of the individuals who were subjected to the intervention. Control group includes individuals who have very similar characteristics to those of the individuals in the treatment group, only they did not experience the impact of the intervention. The outcomes achieved by the control group allow to identify the outcomes that would have been achieved by the treatment group without the intervention. The picture reflects that the individuals selected for both groups (encircled in the picture) are similar (the similarity is reflected by bounding the most similar individuals by vertical dashed lines), except for their participation in the intervention, due to which the treatment group is expected to achieve better outcomes (better outcomes are reflected by portraying the individuals who received assistance higher in the picture than those who did not).
Treatment group (individuals who received assistance, P) and control group (individuals who did not receive assistance, N) (the individuals selected for treatment and control groups are encircled)
As it is suggested in the European Commission‘s methodological documents, counterfactual impact evaluation methods usually encompass double difference (or difference-in-difference) analysis, randomized selection of subjects, propensity score matching and instrumental variable analysis. Also, a combination of methods can be applied. For example, propensity score matching can be used for creating treatment and control groups, while double difference analysis/double difference regression can be employed for evaluating the impact of the intervention.
Double difference analysis/double difference regression
For instance, while evaluating the impact of the EU structural assistance on small and medium-sized enterprises (evaluation commissioned by the Ministry of Economy of the Republic of Lithuania), BGI Consulting concluded that the changes in the number of employees in the treatment and control groups of the measure “New opportunities” were very similar in the period of 2007-2009 (i.e. prior the intervention). The differences manifested only during the implementation of the projects. Such evidence confirmed the suitability of double difference analysis for evaluating the impact of the intervention.
The dynamics of the average numbers of employees in the treatment and control groups of the measure “New opportunities”
In the double difference analysis, the treatment and control groups are compared at different time periods. Firstly, the difference emerging between the groups under comparison (the treatment and control groups) is assessed. Secondly, the difference of characteristics emerging over a certain period of time (before and after implementing the measure) is identified. Double difference analysis takes into account that there are some invisible characteristics conditioning the differences between the treatment and control groups. If these characteristics do not change over time, their impact can be eliminated by comparing both groups before and after the intervention.
The logic of double difference analysis is presented in the picture below, reflecting the impact of the intervention on the number of days worked per year by the individuals.Double difference means that two kinds of differences, firstly,difference in time (before and after the intervention), and, secondly,difference between individuals who were subjected to the intervention and those who were not, are assessed. The picture shows that the average number of days worked per year by the treatment group (individuals who were subjected to the intervention) has changed (increased) over theanalysed period. The average number of days worked per year by the control group has decreased. These are the differences in time (“before-after”). By subtracting the average number of days worked per year by the control group from the average number of days worked per year by the treatment group, a double difference estimator (intervention’s impact estimate) is obtained. That is, double difference analysis is based on the assumption that, in the absence of the assistance, the average number of days worked per year by the treatment group would change equally to the average number of days worked per year by the control group (counterfactual “what would have happened in the absence of the intervention” situation is reflected by the dashed line).
The logic of the double difference analysis
Double difference regression
Double difference regression allows to assess whether the impact is statistically significant. Moreover, it allows to take into account the influence of additional periodic or structural factors.
According to the European Commission’s methodological document, it is reasonable to develop the following regression model:
Yi,t = a + b1*Ti + b2*Pt + b3*Ti*Pt + ԑi,t
where:
Yi,t is the outcome of the person i (e.g. the number of days worked by the person / the fact of being employed at least a certain number of days per year (1 or 0) / person’s income) in the period t;
Ti is a variable with two possible values: 1, if a person participated in the intervention, and 0, if a person did not participate in the intervention;
Pt is a variable with two possible values: 0 reflects the period before the intervention, and 1 reflects the period after the intervention;
Ti*Pt is the outcome of the two aforementioned variables which acquires value 1 only when the outcome of the person who participated in the intervention, in the period after the intervention, is taken into account;
ԑi,t is an error of the regression;
a, b1, b2, b3 are parameters of the regression that are under evaluation:
parameter a reflects the average outcome (e.g. income) of the persons who did not participate in the intervention, in the period before receiving the assistance;
parameter b1 reflects the initial difference between the treatment and control groups;
parameter b2 reflects the difference of the outcomes (e.g. income) of the persons who did not participate in the intervention between periods;
parameter b3 reflects an impact estimate.
At the same time, the significance of the parameters is evaluated. For example, in the cases where the parameter reflecting the impact estimate turns out to be statistically insignificant, it is important to refrain from strong statements regarding definite impact of the assistance. For developing such regression models, econometric packages, for example, free software R (https://www.r-project.org/), are employed.