|In a Nut Shell The Impact That Never Was: The Role of a Control Group in Project Impact Assessment|
|By Henry Mwololo
Development projects continue to play a fundamental role in the livelihoods of rural populations in developing countries. Being largely donor funded, such projects are in most cases social investments aiming to unlock the potential of rural populations. By and large, social investors do not require financial profits to accrue to the investor but rather social benefits to accrue to the target population. Several indicators have been used to measure social benefits including; increased income, increased trade, reduced incidences of disease and reduced cost of doing business. At best, these indicators stop at measuring project outputs and outcomes. The concept of project impact especially at household level remains blurred with many project teams struggling to articulate the impact of the projects they manage.
Literature on project management defines impact as an intended and expected change from a development project as a result of disrupted status quo. In line with this definition, it is worth noting that project outputs and outcomes like increased income among the rural poor are just but a means to an end which would be described as ‘human aspirations’. Humanity aspires for basic needs including; adequate and quality food, clothing, shelter, health and education. It would then be in order to conclude that, any project that does not clearly outline how its indicators at output and outcome levels (e.g. higher income from farming) link with at least one of the human aspirations (e.g. access to improved shelter) should not include an impact component in its logical framework simply because it has none. Articulating, evaluating for and reporting project impact is the bone of contention in the current world of development. As the donor community demands for precise project results (outputs, outcomes and impact) from their investments, the importance of project M&E has increased but without a commensurate improvement in the way M&E teams capture, measure and report project impact.
Several challenges face the process of assessing impact at project level. They include: i) confusing outputs and outcomes with impact hence the lack of project impact from the onset of the project, ii) inadequate M&E skills by
|project team, iii) poor monitoring plan leading to incoherent data hence inaccurate evaluation, and iv) difficulties in attributing outputs and outcomes to a project due to lack of appropriate control group as target population interact freely with the world outside the project. While a wide range of solutions exist for each of these challenges, this article endeavors to give insights into addressing the challenge of attribution of project impact to a particular project using a control group.
A control group refers to a sample that mirrors the project sample in most of the socio-economic aspects being studied. If well identified, the only significant difference between the two samples is the intervention through the project. The control group acts as the counterfactual because it is not possible to observe the ‘project group’ with and without the intervention after a period of time. If the two samples are randomly selected from the same population, and the sample is large enough, any errors would be normally distributed and hence the earlier assumption that any significant differences between the project and control samples are due to the project intervention applies. If the project and control samples are incorporated into the project design, it becomes an experiment referred to as Randomized Controlled Trial (RCT). In case of RCT, the project team should ensure that the control sample does not interact with project activities during the project period to minimize cases of contamination. The project team should collect data regarding the indicators of interest from the two samples at the onset of the project as the baseline. Project implementation should then begin with continuous monitoring and midterm evaluation for any necessary adjustments.
To measure the impact by the end of the project period, follow up data should be collected on the same parameters as were captured during baseline. With the two sets of data, the impact of the project is the difference in magnitude of a particular parameter e.g. income between project and control samples. The project can also assess the magnitude of spillovers which should then be added to the direct project impact. This method of determining impacts is known as the double difference approach with the first one being within the samples and the second between the samples. One of the strengths of this method is that it uses ‘before’ and ‘after’ periods of the project to determine impact ‘with’ and ‘without’ the project.
Erroneously, most development projects fail to factor in a control group at the onset of the project. To address the challenge of lack of a control, scientists have developed methods of artificially
| constructing control groups from within the population and using likelihoods to determine individuals that mirror the project group and hence would have participated in the project had they been given the chance. Indicators from project group are compared to the same indicators from the ‘artificial control group’ and any significant differences statistically tested. One of those methods of constructing control groups where they lack is the Propensity Score Matching (PSM) approach. Any differences between the project group and the constructed control group can confidently be attributed to the group.
In conclusion, project teams should decide the level to which their project delivers. These levels can be output, outcome, impact or a combination. A point worth noting is, while a project can directly deliver an output as well as an outcome, it can only contribute to an impact. In case the project endeavors to contribute towards a societal impact such as access to improved health care, a control group is essential. The project should work with other institutions to substantiate their contribution to the particular impact. Secondly, any differences between the project and control groups should be statistically tested before giving credit to the project interventions. Fair reporting is required especially for those indicators that are not statistically different between the two groups. Lastly, change takes time depending on the dynamics of the target society. Most projects take two and five years. It is feasible to realize significant outputs and outcomes within such a period. However, it is difficult to realize any meaningful impact within the same period of time. Any project intending to contribute to and measure impact should have terminal project evaluation and a post terminal evaluation long after a project closes to allow the target population adjust to the ‘disruption’ the project causes. In case a project does not have adequate resources to plan, monitor, measure and report its contribution to a societal impact, it should predict expected impact instead.
To ensure that projects measure and report accurately the results they deliver including impact, they require teams that are equipped with sufficient tool and skills. Organizations should build the capacity of their M&E teams in order to deliver the ambition of the donor community which is value for their social investments while improving the livelihoods of the societies they serve.
The author is an Impact Assessment - Knowledge Management Specialist and a PhD student (Agricultural Economics) at the University of Nairobi.
|KILIMO TRUST: regional solutions to local problems:|