Blog “Using Data and Evidence in the JICA-supported project in education: The Case of the Project of Developing Mathematics Textbooks in El Salvador”

At the JICA Ogata Sadako Research Institute for Peace and Development (JICA Ogata Research Institute), researchers with diverse experience and backgrounds are forging partnerships with a wide range of stakeholders and partners. In this blog post series, we share their knowledge and perspectives gained from their research activities. This time, Maruyama Takao, senior research fellow at the institute, discusses how data and evidence were used in a JICA-supported project to develop mathematics textbooks in El Salvador.

Author: Maruyama Takao, senior research fellow, JICA Ogata Sadako Research Institute for Peace and Development

JICA has conducted technical cooperation to develop mathematics textbooks in Central America since the early 2000s. As the textbooks are important intermediate that links curricula, teachers, and students, it is possible to improve teaching and children’s learning by the textbook development. Nishikata (2017) describes the history of JICA’s technical cooperation to develop mathematics textbooks in Central America. (Available in Japanese only.)

Nishikata,　 Chubei no Kodomotachi ni Sansu/Sugaku no Gakuryoku Kojo wo: Kyokasho Kaihatsu wo Tsuujita Kokusai Kyoryoku 30 nen no Rekishi (tentatively translated as Enabling Better Mathematical Performance of Children in Central America: Three Decades of International Cooperation Through Textbook Development), 2017.

In the 2010s, various studies demonstrated the low level of learning outcomes among children in developing countries (Davidson & Hobbs, 2013; Altinok et al., 2014). The importance of learning outcomes for the economic growth of countries were also argued in the research (Hanushek & Woessman 2012; 2015; 2016). In the context, the Sustainable Development Goals clearly mentioned of improved learning outcomes for literacy and mathematics among children in the target 4.1.

JICA’s Project for the Improvement of Mathematics Teaching in Primary and Secondary Education (the ESMATE project) started in El Salvador in 2015. The ESMATE project focused on the improvement of children’s learning in mathematics and impact evaluation was conducted (Nishikata, 2017; JICA, 2019).

Besides the understanding of national curriculum, authors of mathematics textbooks should have a good understanding of the status of mathematics learning among children in the country. When the ESMATE project began, the Ministry of Education in El Salvador conducted an assessment on student learning levels in mathematics. However, the test items were too advanced to measure student academic performance. With the advice of JICA experts, a team in charge of developing textbooks at the Ministry of Education (the MINED team) prepared mathematics test according to the national curriculum. The results of the assessment revealed the low level of academic performance for mathematics among students in El Salvador (JICA, 2019). The results built common understanding between the JICA experts and the MINED team to develop textbooks.

During the course of textbook development, trial of classes using draft versions took place at several pilot schools. To see whether the classroom teaching was organized and students learned as intended during the textbook-development process, the MINED team visited these schools to observe the class and conduct interviews with teachers. Based on the findings from these schools, the textbooks were further revised.

As teachers in El Salvador organized mathematics classes without sufficiently checking the student notebooks and their answers, their behavior might not change by just distributing the newly developed textbooks and offering introductory training to teachers. Therefore, review meetings were introduced for teachers to reflect upon their teaching using student math test results. At the meetings, test results (such as mean scores) were shared among the participants, and they discussed how to improve student mathematics learning. The process allowed teachers to improve their teaching methods.

Does the intervention package including the distribution of mathematics textbooks developed by the ESMATE project (“ESMATE program”) improve student learning outcomes in mathematics? To answer the learning agenda, RCT targeting primary second grade students were conducted. In RCT, intervention is randomly assigned to the subjects or units in the study, and the treatment and the control groups are compared to rigorously evaluate the impact of the intervention.

Table 1 shows the RCT design in the ESMATE project. From the departments that were covered by the study, 250 schools were randomly selected. (Note: El Salvador consists of 14 administrative divisions called departments.) Half of the sampled schools were randomly assigned to the treatment group, and the other half were assigned to the control group. In the 2018 school year, which was the first year of the study (Year 1), the ESMATE program was provided to the treatment group but not to the control group. In the 2019 school year, which was the second year of the study (Year 2), the control group received the program as well.

The results from the end of Year 1 show the one-year impact of the ESMATE program. Students were tracked over two school years. We would be able to see the accumulated impact of the first-year interventions in the following year, if the learning outcomes of the treatment group remained better than those of the control group as of the end of Year 2.

Fig. 1 shows the distribution of mathematics test scores for students at the end of Year 1. The red solid line represents test-score distribution in the treatment group while the blue dotted line represents test-score distribution in the control group. Score distribution in the treatment group is higher than the control group.

As the curriculum differs in each grade, the different test items were used at the surveys in 2018 and 2019. By simply comparing raw scores, it is hard to capture the academic progress from Year 1 to Year 2 as the math tests were different. Therefore, based on the common five items in the two tests, the test scores in Year 2 was rescaled to those in Year 1 using the item response theory. At the end of Year 2, as shown in Fig. 2, although the difference in math scores between the two groups became smaller, the math scores in the treatment group remained higher than the control group. This confirmed that students in the treatment group advanced math learning in Year 2 based on the improved understanding in the previous year.

In El Salvador, as a result of the presidential election held in February 2019, a change of government was on its way. Education policies were often drastically changed following a change of government in Latin America. Developed textbooks might not be used when textbooks policies were to be changed. The ESMATE project contacted the new Minister of Education to gain her understanding on the effectiveness of the ESMATE program, referring to evidence from the RCT.

In May 2019, the Ministry of Education in El Salvador held a seminar to share the achievements in the ESMATE project, including evidence, with local actors in the country. The Ministry explained the importance of efforts to improve mathematics learning while presenting assessment results, and the Ministry demonstrated the effectiveness of the ESMATE program, referring to evidence.

The communication using evidence supported the continuous use of the textbooks developed in the ESMATE project after the government change in 2019.

In my previous blog post, I presented a search, learning, and communication cycle using data and evidence from the case of the Indian NGO, Pratham. While a similar cycle was observed in the ESMATE project, it was an ad-hoc one as the ESMATE project was implemented with a fixed period for JICA’s technical cooperation. From 2021, a project aiming to establish an assessment system for formal mathematics education began in El Salvador. This can be regarded as a project aiming to institutionalize the cycle in El Salvador.

Since JICA is operating different projects in developing countries, JICA should link the search, learning, and communication cycle in El Salvador with the JICA-supported projects in other countries to advance the organizational learning.

Altinok, Nadir, Claude Diebolt and Jean-Luc Demeulemeester. 2014. “A new international database on education quality: 1965–2010.” Applied Economics, 46.

Davidson, Marcia, and Jenny Hobbs. 2013. “Delivering reading intervention to the poorest children: The case of Liberia and EGRA-Plus, a primary grade reading assessment and intervention.” International Journal of Educational Development, 33(3): 283–293.

Hanushek, Eric L., and Ludger Woessmann. 2012. “Schooling, educational achievement, and the Latin American growth puzzle.” Journal of Development Economics, 99 (2): 497-512.

Hanushek, Eric L., and Ludger Woessmann. 2015. The Knowledge Capital of Nations: Education and the Economics of Growth. MIT Press, Cambridge.

Hanushek, Eric L., and Ludger Woessmann. 2016. “Knowledge capital, growth, and the East Asian miracle.” Science, 351: 344-345

独立行政法人国際協力機構 (JICA). 2019. エルサルバドル共和国初中等教育算数・数学指導力向上プロジェクト業務完了報告書.

Maruyama, Takao and Takashi Kurosaki. Forthcoming. "Developing Textbooks to Improve Math Learning in Primary Education: Empirical Evidence from El Salvador." Economic Development and Cultural Change. https://doi.org/10.1086/721768　

Maruyama, Takao. 2022. "Strengthening Support of Teachers for Students to Improve Learning Outcomes in Mathematics: Empirical Evidence on a Structured Pedagogy Program in El Salvador." International Journal of Educational Research, Vol. 115: 101977. https://doi.org/10.1016/j.ijer.2022.101977　

西方憲広（2017）『中米の子ども達に算数・数学の学力向上を』佐伯印刷 （JICAプロジェクトヒストリーシリーズ）.

Note: This blog expresses the individual views of the author, not the views of JICA nor of the JICA Ogata Sadako Research Institute for Peace and Development.

About the Author:
Maruyama Takao is a senior research fellow at the JICA Ogata Research Institute since 2022. He joined JICA in 2002 and has worked in different sections including the Africa Department, Senegal Office and Human Development Department.