June 23, 2016
The JICA Research Institute (JICA-RI) held the Open Seminar on "Current Trends and Future Challenges in Impact Evaluation around the World" on May 24, 2016, at the JICA Ichigaya Building.
Emmanuel Jimenez, Executive Director of International Initiative for Impact Evaluation (3ie) - established in 2008 to promote evidence-informed development policies and programs - gave the keynote speech "Current Trends and Future Challenges in Impact Evaluation in Developing Countries." JICA-RI Visiting Fellow Yasuyuki Sawada (a professor at the University of Tokyo) and JICA Evaluation Department Deputy Director General Satoshi Shigiya commented. Approximately 70 people gathered and a lively question and answer session took place.
Jimenez first gave the example of a World Bank financed project on improving health in Russia in which he was involved, and talked about a problem that people involved in the project tended to confront. Since the project began, abortions began to decline and people took this to be an achievement of the project. However, in reality, it turned out abortions already started declining before the project and the same trend could be seen in other regions, so it could not be said definitively that this was a result of the project. Jimenez defined impact evaluation as "the study of changes in outcomes that are attributable to an intervention," and he said it is a method that needs to be adopted.
Regarding methods of impact evaluation, Jimenez introduced the concept of Randomized Controlled Trials (RCTs) in which regions and groups are selected from the population, and these members are randomly assigned to be either beneficiaries who receive the project intervention (the treatment group) or non-beneficiaries who receive no intervention (the control group). After the project has been implemented, the outcome indicators of each are compared. He also introduced the concept of a quasi-experiment such as "matching" in which a group with characteristics similar to the project participant group is chosen from among nonparticipants.
According to an investigation by 3ie, the fewer than 50 cases of impact evaluation in the world in 2000 had grown to nearly 400 cases by 2012. But because most of them have been in India, South America, Kenya and South Africa and in the four fields of 1) health, nutrition and population, 2) education, 3) social protection and 4) agriculture, Jimenez said an issue in the future will be implementation in regions and fields that are not currently covered.
Jimenez pointed out another issue as well: despite an impact evaluation demonstrating great results, projects sometimes end without being expanded, and the results of impact evaluations sometimes are not cited in a project's final report. He mentioned possible reasons for this, including that researchers and implementers may have different points of view and that evidence-based policy decisions have not yet become common.
Commentator Sawada explained that as we entered the 21st century, development economics began to receive attention as a major field in economics, and as part of that process there has been an impact evaluation revolution based on RCTs, and field trials that experimentally investigate the effects of policy in the real world have become a major tool in positive economics. On the other hand, he brought up another problem facing impact evaluation: implementing it in fields like infrastructure and governance where it is difficult to evaluate impact.
Shigiya gave a detailed explanation of evaluation at JICA. He said 17 impact evaluation reports have been published since 2008. They can be broken down by sectors as follows: six on agriculture/irrigation, five on education, two on health, two on forests, one on small-scale infrastructure and one on water. With regard to regions, there were 10 in Africa, five in Southeast Asia, one in the Middle East and one in South Asia. In addition, more than 10 are currently underway. There are more than 200 new projects per year and only a limited number of impact evaluations are performed. Shigiya explained this by saying that compared to a post evaluation, an impact evaluation requires specialized knowledge of econometrics and statistics, and investigations are complex and high cost. He raised the following possible solutions: encouraging the participation of consultants and researchers, and developing impact evaluation methods that use existing data. Regarding the gap between researchers and practitioners, he said both JICA-RI and the JICA Evaluation Department should mediate between them.
Richard Manning, former OECD-DAC chair and current chair of 3ie's Board of Commissioners, closed the seminar by saying, "I'd like to see all agencies always have the questions, 'What is the basis?' 'What have you learned from the evaluation before you came into the project?'"
