History and New Perspectives on ODA Evaluation

ODA evaluation in Japan began in 1975 when OECF (Overseas Economic Cooperation Fund, a predecessor of JICA) started conducting ex-post evaluations of yen loans. ODA has placed a strong emphasis on project evaluation since an early stage, ahead of other administrative agencies. The goal was to properly manage individual projects and make them more effective by conducting ex-post evaluations. In addition, project evaluation was required to summarize the results, challenges, and lessons learned, and report them externally to fulfill "accountability", and to "utilize lessons learned (improve)" for future projects. The role of ex-post evaluation is to clearly demonstrate the success or failure of a project based on clear evidence.

The impetus for addressing "accountability" and "utilizing lessons learned" dates back to the 1980s, when criticism of ODA was at its peak. Jin Sato, a professor at the University of Tokyo specializing in development cooperation, notes in his book "Formation of Development Cooperation: Ecological Perspective on Self-reliance and Dependency" that "the criticism of ODA in the 1980s was directed at the agencies that implemented ODA that had material to rebut, but failed to take full advantage of the opportunity, and as a lesson from this, the accountability of those implementing development cooperation has been called into question." As a result, ex-post evaluation was implemented not only for project management, but also for accountability and utilizing (improving) lessons learned.

JICA is also attempting to adopt ex-post evaluation methods that reflect the times. An example of this is the Quick Review of the Emergency Support Loan for Crisis Response to New Coronavirus Infection conducted in fiscal year 2023. Unlike ex-post evaluations for individual projects, this review aims to provide an overview of the results and suggestions obtained at this point, so that timely feedback can be given for similar future projects.

JICA conducts its project evaluation based on the international perspective of ODA evaluation by the Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development (OECD). The DAC evaluation criteria were revised in 2015 following the adoption of the SDGs. JICA also revised its project evaluation criteria, incorporating new perspectives such as "Leave No One Behind" and "People's Well-being" from the SDG perspective. In the "JICA Annual Evaluation Report" released in April of this year, new evaluation perspectives that focus on "Leave No One Behind" for those at high risks of being left behind during the project planning and implementation stages and "People's Well-being" which is to ensure that people's lives, dignity, and well-being are protected and fulfilled, are introduced using specific examples. By incorporating these perspectives, JICA aims to establish an evaluation practice that is in line with its mission of "Human Security."

In 2022, JICA organized the perspective on "Project Evaluation in Conflict-Affected Countries." As it is not easy to conduct ex-post evaluations of projects implemented under the influence of internal conflicts or political upheavals, this is a perspective for organizing the indicators when data acquisition is difficult and for conducting post-project evaluations of projects that experienced large-scale unrest during the project implementation stage that was not anticipated in the planning stage. It is important to "document" the decision-making process and rationale for changing the project scope based on the history and changing circumstances of continuing the project in the midst of political turmoil.

Professor Sato Jin points out in his book that there are projects that are considered to be successful because they are effective many years after completion, even if they are criticized for being ineffective at the time of implementation. In the field of development cooperation, a positive effect that was not initially anticipated can be produced and continue to have a positive impact on the local community. This shows that societal and personal consciousness changes often take a long time and predicting these changes is difficult.

The Japan International Cooperation Agency (JICA) is working to objectively grasp the medium and long-term outcomes of its projects based on evidence. First, they collected and organized papers on impact evaluations conducted both inside and outside JICA to understand the outcomes of past development cooperation projects and to be used for future analysis. They are also making efforts to use satellite data as one means of quantitatively understanding the effects of projects at the time of post-evaluation. Furthermore, they used a new household survey method called SWIFT (Survey for Well-being via Instant and Frequent Tracking) developed by the World Bank to monitor and understand the effects of the "Market-Oriented Small-scale Horticulture Promotion Project" in Malawi . SWIFT has fewer survey items compared to traditional household surveys, and because it uses electronic devices to collect data, it is characterized by being able to obtain data inexpensively, quickly, and easily. Using the characteristics of SWIFT, they confirmed the impact on the household expenditure of farmers who participated in the technical cooperation project, and whether the farmers changed their behavior from "making and selling" to "making for selling".

JICA has a responsibility to use various methods to explain outcomes easily. "Ease of understanding" is a critical element of its responsibility.

Many people may instinctively react to the word "evaluation". However, evaluation is a "tool" and a "perspective" for improvement. Through my work in evaluation, I have been able to deepen my understanding of international cooperation and realize new perspectives and ways of thinking. The Evaluation Department will continue to work to deepen understanding and raise interest in evaluation, make ODA more familiar to everyone, and further improve projects using evaluation.