Ex-post Evaluation

1.1 Background of the Project

In Mongolia, the Extended Programme on Immunization (EPI) has initiated based on existing immunization activities starting since 1962. The government of Mongolia launched specific disease control initiatives in 1993 for EPI targeted diseases with assistance from international organizations (UNICEF, WHO) and had achieved high immunization coverage. However, the government of Mongolia had difficulties in self-reliance in EPI.

On the other hand, according to the result of various surveys in 1992 and 1993 conducted by the Government with assistance of UNICEF, Iodine Deficiency Disorder (IDD) was acknowledged as a serious problem in Mongolia.

From these points of views, in order to promote maternal and child health, the government of Mongolia requested the government of Japan to launch a project on technical cooperation. The government of Japan responded to the request and implemented the Project in October 1997 to eliminate IDD and to enhance the quality of EPI.

1.2 Project overview

The Maternal and Child Health Project (further the Project) was implemented October 1997- September 2002. In June 2002, Japanese Evaluation Team visited Mongolia and conducted a Final Evaluation jointly with the Mongolian side. The purpose of the evaluation was to evaluate implementation and achievements of the Project. During the evaluation, the Joint team reviewed all activities and achievements of the Project and evaluated the project by using DAC five criteria namely efficiency, effectiveness, impact, relevance and sustainability. Both sides concluded that at the end of the project (only 3 months after the evaluation) project would achieve its purpose.

In January 2006, JICA Country Office has decided to conduct Ex-post Evaluation on Maternal and Child Health Project (further Project) and called for external evaluators. The Mongolian Public Health Professionals’ Association (further MPHPA) has sent its proposal to JICA and was selected as an Evaluator.

(1) Overall Goal

To promote maternal and child health in Mongolia

(2) Project Purpose

1) To eliminate IDD
2) To achieve self-reliance in the EPI

(3) Outputs

1) IDD Elimination Program

• National IDD laboratory is established
• All the salt factories produce iodized salt
• All the salt on the retail level is iodized and purchased by consumers
• Knowledge, Attitude and Practices (KAP) of the people about the importance of using iodized salt is enhanced
• Referral system for monitoring the progress of IDD elimination is established
• National IDD Program becomes self-sustainable

2) EPI

• Reliable clinical surveillance system is established
Reliable cold chain is established
• Willingness for vaccination is enhanced

(5) Inputs (at the time of Project Termination)

Japanese side:

Mongolian side:

EPI project

IDD Elimination project

3.1 Summary of Evaluation Results

IDD Elimination project:

(1) Impact

(2) Sustainability: IDD Elimination Project

Policy aspect:

Since 1997, the government of Mongolia has been changed several times. However, the government’s commitment to implement this Program is maintained until now. 2006 is the last year of the second National Program to Control IDD, therefore the Government of Mongolia is planning to discuss the future of it during a National Workshop to be held in March or April 2006. It is expected to continue but its ownership might change. The Association of Salt Manufacturers, established by the joint initiative of UNICEF, JICA and MOFA, is expected to take responsibility for further implementation of the National Program.

Organization aspect:

Capacity built at institutional level ensures sustainability of the project in the near future.

Financial aspect:

In order to ensure sustainability of management and coordination of the National Program, the government allocates 15mln MNT to MOFA and 5mln MNT to MOH annually. It helps to ensure sustainability of management and coordination of IDD elimination and salt iodization activities.

Local governments also implement sub-programs and fund it from their local budget in order to ensure the Program success in their respective aimags.

EPI project:

(1) Impact

(2) Sustainability: EPI Project

Policy aspect:

Government is highly committed in implementing EPI program, and it has included vaccination issues in goals of many of its policy documents, including Millennium Development Goal, Poverty reduction strategy etc.

MOH in collaboration with donor agencies is developing Multi-year plan for National Immunization Program, which identifies actions to be taken until 2010. It will also ensure sustainability of the Project achievements.

The government also aims to improve self-reliance of EPI program by improving Vaccine Fund. It is expected that by 2010 the government becomes self-reliable and buys all the vaccines necessary for Mongolia by itself.

Organization aspect:

Partnership between MOH and international organizations especially JICA and UNICEF was perfect. They cooperate openly, share ideas and divide responsibilities and work on different elements of the Program so support each other. This collaboration continues at present and other donors that enter in this collaboration also become part of this well organized collaboration and works in good partnership.

Surveillance system for EPI target diseases is maintained at high level of performance. National Polio Laboratory and National Measles Laboratories are working in full capacity and ensure confirmation of disease cases by Lab diagnosis.

Financial aspect:

MOH is receiving supply of EPI vaccines by other donors. Top donor is JICA, if JICA stop supply, 60% children of Mongolia can’t be vaccinated immediately. By Master Plan of EPI planed on Oct 2005, MOH will achieve independence on 2010 completely. Therefore, financial aspect is not secured sustainability at the present moment.

3-2. Factors that have promoted project

IDD Elimination project:

Some other programs and projects implemented by MOH with assistance of other international and bilateral organizations, support sustainability of IDD Elimination Programme. In addition, some policy decisions of the government also support implementation of IDD elimination program. The government of Mongolia pays special attention on food fortification especially fortification of flour with iron, milk with vitamin calcium and others. There are some policy initiatives to make salt iodization a part of food fortification project in order to ensure sustainability of achievements of IDD program.

The government has established Trade Networks in several aimags and identified main food products, which must be supplied through this network. Iodized salt is included among these products and regularly supplied to aimags, which are covered by these networks.

The government of Mongolia also announced that it would support private sector development. Therefore, small-scale salt factories, which are part of private sector, have better chance of survival.

EPI project:

Vaccination is included in work plan of every level governors and understood by most of the policy makers and decision makers, therefore, although government officials are changed frequently, even new ones are also committed for improving vaccination coverage.

3-3 Factor that have inhibited project

IDD Elimination project:

There are several factors that negatively influence on achievements of IDD elimination and salt iodization.

Some small salt deposits are not owned by anyone, and local people freely use natural non-iodized salt from salt deposits nearby. Moreover, some of the natural salts are not suitable for human but still used by local people.

According to currently effective legislation, tax on salt deposits is equal to tax of gold mining deposits. If this situation is not changed, nobody will own and extract salt from small salt deposits and it will negatively influence on sustainability of the project.

EPI project:

High turnover and staff mobility negatively influence on sustainability of the project, because due to these previously trained staff are changed or move away and new staff needs to be trained again.

Frequent natural calamities sometimes negatively influence on the vaccination coverage because due to harsh winter and draught people move from their original living places to other aimags where they are not registered and cannot access health services Including vaccination.

3-4. Conclusions

IDD Elimination project:

Although some of the goals of the IDD Elimination Project implemented by JICA have not been reached yet, in general, project has significantly contributed to Control of IDD in Mongolia. On the contrary, no negative impact was observed.

Enabling policy and legislative environment, existence of adequate human and other resources, involvement of private sector and government’s commitment to maintain project achievements are the factors, which ensure sustainability of IDD Control Programme in Mongolia.

EPI project:

On overall, EPI Project implemented by JICA 1997-2002 has successfully achieved its goals and greatly contributed to strengthening EPI Programme in Mongolia.

Enabling policy and legislative environment, existence of adequately trained human resource, well-structured infrastructure, promising financial long-term funding mechanism and government’s commitment to maintain project achievements are the factors, which ensure sustainability of EPI Project in Mongolia.

3-5. Recommendations

Regarding IDD Control Program

Utilization of iodized salt is still lower in western region’s aimags compared to other regions. Therefore, the government should take actions aimed to improve access to iodized salt in western aimag, decrease price of iodized salt and control use of non-iodized salt by local people.

Knowledge and attitude of general population relies on frequency of information given to them. Therefore, in order to sustain achievements of the Project, appropriate funding should be allocated for IEC activities directed for the general population.

There is a need to pay special attention on mobility of trained health professionals and need to develop comprehensive system-wide human resource development policy aimed to ensure sustainability of trained work force of health sector especially in rural area.

In order to promote exploitation of salt deposits by local businessmen, taxation on ownership of small salt deposit should be very low or temporarily free of tax. Currently, tax on salt deposit exploitation is equal to gold deposit tax.

There are around 90 rivers and lakes from which local people get non-iodized salt and utilize for cooking purposes, when they are unattended and guarded. Therefore, local governments need to pay attention on ownership of natural salt deposits and prevent local people from using non-iodized salt from their local areas.

Regarding EPI Program

Cold chain equipment provided by JICA, UNICEF and WHO are becoming old and requires maintenance. Therefore, government should pay attention on training qualified repairmen and allocating budget for purchase of necessary spare parts and maintenance of equipment.

In order to operate the National Vaccine Fund efficiently, Mongolia needs to work with step-by-step plan and to seek ways to buy vaccines cheaper by using contacts of international partners.

3-6. Lesson learned

The IDD elimination project evaluation is mostly based on data of 1st (1992), 2nd (2000-2001) and third (2004) National Nutritional Surveys. However, 2nd Survey data is data of 2000-2001 and third Survey data are data of 2004. It means they are not actual figures corresponding to the time of completion (Sep 2002) and impact 3 years of after the completion (2005). It is advisable to have conducted third National Survey in 2005 exactly 3 years after the completion of the project in order to get true picture.

These 3 consequent surveys sometimes used different methods and indicators, which makes comparison difficult. For instance, to show iodine content in urine, Survey II used arithmetic mean while Survey III used median. Therefore, attention should be paid to use the same method and indicators.