Iterative implementation of Impact Evaluation and the Utilization of Their Findings in the JICA “Ecole pour tous (EPT)” project in Madagascar

At the JICA Ogata Sadako Research Institute for Peace and Development (JICA Ogata Research Institute), researchers with diverse experience and backgrounds are forging partnerships with a wide range of stakeholders and partners. In this blog post series, we share their knowledge and perspectives gained from their research activities. This time, Maruyama Takao, senior research fellow, wrote this blog post to provide an overview of iterative implementation of the impact evaluation in Madagascar and discuss some of the lessons learned for future evidence use.

Author: Maruyama Takao, senior research fellow, JICA Ogata Sadako Research Institute for Peace and Development

With a focus on school management committee (SMC) that consist of representatives of school and local community, JICA has supported the improvement of education through collaboration between school and local community by democratically establishing the committee and enhancing their capacities since the early 2000s. This project is called Ecole pour Tous (EPT; also known as “School for All” in English) and has been operated in multiple countries in Sub-Saharan Africa, including Niger and Madagascar (Details of the project can be found in the links below. Available in Japanese only).

To meet the various educational needs of local community, a model for collaboration between school and local community and the packages of interventions including a series of trainings were developed in the EPT project. The impact evaluation was conducted for the packages as a research project of the JICA Ogata Research Institute (Sawada et al., 2022; Kozuka, 2023). In this post, I would like to share three cases of impact evaluation that were conducted in response to the evolution of the project activities in Madagascar from 2018 to 2022 and how evidence was used.

In Madagascar, after the EPT project began in June 2016, the project introduced a method for literacy and numeracy teaching called “Teaching at the Right Level (TaRL)” that was developed by the Indian NGO Pratham. Through several pilots from 2017 to 2018, an intervention package called PMAQ-TaRL was developed. This is made up of two components: school management improvement component and pedagogy component.

The school management component offers training for school principals and SMC bureau to build capacity to organize democratic establishment of SMCs; participatory development and implementation of school action plan cycle; and resource management. The pedagogy component consists of TaRL training for teachers and community volunteers and distribution of teaching and learning materials. With the improved capacity, SMC develops and implements an action plan that includes remedial activities using TaRL through community-wide collaboration.

While PMAQ-TaRL was developed in mid-2018, is this intervention package really effective in improving literacy and numeracy of children? The project conducted surveys on learning outcomes in pilots; however, the study was simple comparison of before-and-after study and unable to identify the impact in a rigorous manner. To investigate the above question, a randomized controlled trial (RCT)* was conducted in the Amoron’i Mania region (AM) in South Central Madagascar in the 2018-2019 school year. Out of approximately 1,000 schools in AM, 140 were randomly selected and half of these (70 schools) were assigned to the treatment group (i.e., subject to PMAQ-TaRL), and the other half was assigned to the control group (i.e., no intervention). Results of RCT demonstrated that the package realized remedial activities using TaRL and improved foundational literacy and numeracy skills of children (Maruyama & Igei, forthcoming).

*In an RCT, an intervention is randomly assigned to a part of the subjects of the study. The treatment and the control groups are compared, allowing the impact of the interventions to be rigorously evaluated.

The Ministry of Education in Madagascar scaled up the school management component in the 2019-2020 school year and the pedagogy component for literacy in the 2020-2021 school year for the remaining approximately 900 schools in AM. In the scaling-up phase, the training implementation structure must be changed from the pilot phase. Cost should be also reduced. Would the intervention package whose impact was verified during the pilot phase be able to improve learning outcomes in the scaling-up phase? To explore this question, a second round of impact evaluation was conducted after school closures due to the COVID-19 pandemic were lifted. This study covered a total of 130 schools in an area at the borders of neighboring regions of AM and the Haute Matsiatra (HM) and used difference in differences (DD).

As interventions on the school management component were conducted in the 2020-2021 school year in HM—one school year later than in AM—the impact of the pedagogy component following improvements in school management was able to be estimated by comparing the two regions using DD. Although the pedagogy component improved children’s reading skills, a challenge remained in learning improvement from word-level reading to reading short sentences (Maruyama & Igei, 2023a). Based on the impact evaluation results, possible causes behind this challenge such as the shortage of teaching materials were considered, and measures to address them were taken when scaling up the package to other regions.

In the following school year (2021-2022), the Ministry of Education in Madagascar scaled up the pedagogy component for numeracy on the AM side of the border and the literacy portion of this component on the HM side. On the AM side, does the scaling-up of the pedagogy component for numeracy have positive impact on children’s mathematical learning? In addition, is the impact on children’s reading skills sustained one year after the scaling-up of the pedagogy component for literacy in this area? Moreover, on the HM side, is learning improvement achieved in children’s reading skills following the scaling-up of the pedagogy component for literacy? To investigate these questions, a third round of impact evaluation took place. We conducted a follow-up survey for the children (grade 3 as of the 2020-2021 school year) who participated in the second round of impact evaluation and evaluated the impacts using DD. On the AM side of the border, the pedagogy component for numeracy improved the learning outcomes and the impact on foundational literacy sustained. On the HM side, the pedagogy component on literacy improved the learning outcomes.

Evidence from the impact evaluation have been widely shared with stakeholders at the Ministry of Education in Madagascar and JICA, supporting the policy of scaling-up of PMAQ-TaRL in Madagascar. On another note, academic papers become publicly available (Maruyama & Igei, 2023a; 2023b; forthcoming) and through communication of the impact of PMAQ-TaRL, global recognition of the JICA-supported project is enhanced.

To conduct impact evaluation and use evidence in the development practice, what lessons can be learned from the case in Madagascar? I would like to highlight the following three points with a focus on practical perspectives.

First, in the Madagascar case, impact evaluation was planned and implemented with regards to strategically important questions (i.e., learning agenda) for the project. As impact evaluation requires technical expertise, the design of the evaluation framework tends to be researchers oriented (Shah et al., 2015). It is therefore crucial to identify key questions that need to be answered in terms of development strategies in practice, through daily dialogues among the development agency, project team members, and researchers.

Second, evaluation results need to be shared and fed back to the practice in a timely manner. In order to practically utilize impact evaluation results, results should be shared in time with decision making on next steps and key events (e.g., meetings of joint coordination committees for the project) in the development practice. As activity plans and budgets are formulated for fixed periods, evaluation results cannot be sufficiently reflected in decision making if the feedback miss the right timing, no matter how great the impact evaluation is. The development agency, project team members, and researchers can deepen learning through discussing tentative results from impact evaluation.

Finally, the development agency staff, project team members, and researchers should have a common set of goals along with clear division of roles and communication. While writing good academic papers and getting them published on academic journals are often goals for researchers, researchers should have a broader common goal with the development agency staff and project team members when collaborating with them. In the Madagascar case, goals—to address the global learning crisis and to support children in the acquisition of foundational literacy and numeracy skills—seem to have been shared between the research and practice sides. When conducting impact evaluation, various matters have to be quickly and smoothly decided between the two sides and then implemented. Clear division of roles and communication can be seen as the basis for this.

In this post, I have shared an overview of iterative implementation and feedback of the impact evaluation in Madagascar and considered the lessons learned for future evidence use. JICA and the EPT project are expected to continue exploring what challenges and questions that have to be addressed and what improvements could be made for their practice using evidence.

Kozuka, Eiji. 2023. “Enlightening communities and parents for improving student learning: Evidence from Niger.” Economics of Education Review, Vol. 94: 102396.
https://doi.org/10.1016/j.econedurev.2023.102396

Maruyama, Takao and Kengo Igei. 2023a. "Scaling up Interventions to Improve Basic Reading: Evidence from Madagascar after the COVID-19 Pandemic Shock on Education." JICA Ogata Research Institute Discussion Paper, No. 4.
https://www.jica.go.jp/jica-ri/ja/publication/discussion/dp_04.html

Maruyama, Takao and Kengo Igei. 2023b. "Developing Collective Impact to Improve Foundational Learning: Evidence from Madagascar After the COVID-19 Pandemic Shock." JICA Ogata Research Institute Discussion Paper, No. 15.
https://www.jica.go.jp/jica_ri/publication/discussion/1517363_24127.html

Maruyama, Takao and Kengo Igei. Forthcoming. "Community-wide Support for Primary Students to Improve Foundational Literacy and Numeracy: Empirical Evidence from Madagascar." Economic Development and Cultural Change.
https://doi.org/10.1086/726178

Sawada, Yasuyuki, Takeshi Aida, Andrew S. Griffen, Eiji Kozuka, Haruko Noguchi, and Yasuyuki Todo. 2022. “Democratic institutions and social capital: Experimental evidence on school-based management from a developing country.” Journal of Economic Behavior & Organization, Vol. 198: 267-279.
https://doi.org/10.1016/j.jebo.2022.03.021

Hara, Masahiro. Nishi Afurika no Kyoiku wo Kaeta Nihon-hatsu no Gijyutsu Kyoryoku: Nijeru de Hana Hiraita EPT Purojekuto (Japanese Technical Corporation that Changed Education in West Africa: Success of the EPT Project in Niger). Project History, Diamond, 2011. (Available in Japanese only.)
https://www.jica.go.jp/jica_ri/publication/projecthistory/20110401_01.html

Japan International Cooperation Agency (JICA). JICA Minna no Gakko no Ayumi (2014-2016) Nyu-su Retaa Geppo-shu (History of JICA’s School for All Project [2014-2016]: Collection of Project Newsletters and Monthly Reports), 2019. (Available in Japanese only.)
https://openjicareport.jica.go.jp/246/246/246_523_1000040612.html

Note: This blog expresses the individual views of the author, not the views of JICA nor of the JICA Ogata Sadako Research Institute for Peace and Development.

About the Author:
Maruyama Takao is a senior research fellow at the JICA Ogata Research Institute since 2022. He joined JICA in 2002 and has worked at the Africa Department, Senegal Office and Human Development Department among others.