Ex-post Evaluation

1-1. Background of the Project

The project "Global Improvement of CHUM" is the first trial tripartite project where the Madagascan Ministry of Health, the French Cooperation and JICA work jointly for the improvement of health services. It is in the framework of Health Collaboration in Africa according to the Frenco-Japanese agreement signed by French President Chirac and the Japanese Prime Minister Hashimoto during the 1996 summit. The tripartite agreement among the Madagascan, French and Japanese Governments was signed on June 2, 1999. The project lasted from 1999 to February 2004.

1-2. Project Overview

(1) Overall Goal

1) CHUM contributes to the improvement of medical care in Mahajunga.

2) Sustainability of the project is ensured by improved hospital management.

(2) Project Purpose

The number of patients that receive medical services increases.

(3) Outputs

1) Increased referral cases

2) Improved patients’ satisfaction with medical services of CHUM

3) Tariffs affordable for patients set by CHUM

4) Improved administrative capacity of CHUM

5) Information on CHUM being disseminated to patients and local communities

(4) Inputs (as of the Project's termination)

Japanese side:

French side:

Madagascar Side:

3-1 Summary of Evaluation Results

(1) Impact

1) Impacts on the overall goal

Statistical data, interviews of stakeholders and CP show that the referral system has a positive impact. This impact has been measured with several stakeholders of the project such as CHUM and other medical facilities in Mahajunga. The Regional Health Office (DRS) confirmed that the impact is generally positive; 3 Basic Health Centers (CSBs) out of 5 responded said that after the termination of the project, the number of referred cases they have sent to CHUM has increased. The last 2 CSBs have noticed that the number of the cases they needed to refer to has decreased much thanks to the training provided by specialist doctors from CHUM to doctors in CSBs.

The Ministry of Health (MOH) finds that the referral system set by the project at CHUM was quite successful. For example, during the project in rural areas, there was real dedication of the communities. The involvement of the local people (through initiatives they took to coordinate between themselves to do what is the necessary to bring the patients referred to CHUM), their will to work together is a really good impact of the project. Actually, the project made them much closer to the health office in the area and more responsible concerning health of the community. After the final evaluation of the project, MOH felt the necessity of spreading this and it is now elaborating a manual, based on CHUM model, which will be dispatched to all health centers throughout the country.

2) Other impacts

CHUM is a university hospital but medical education was not included in the project: at that time, priority was to "raise" the hospital and to improve care. Even if the education was not included in the project, students are also indirectly benefiting from the project. Actually, students improve their knowledge thanks to equipment provided by the Japanese government and through trained doctors supervising and teaching them during the training. Techniques have been transferred to C/P. Students and trainees at CHUM are well aware of the referral system and sustainability can be ensured for those who will be dispatched in the districts. CHUM has received about 600 students in 2003, 300 in 2004 and 400 in 2005.

(2) Sustainability

1) Financial sustainability

The financial sustainability is critical to the sustainability of the project. Unfortunately, the financial capacity of CHUM does not allow it to upgrade its present performance. The receipt of CHUM during the project has always increased due not only to the users' financial contributions (PFU) but also to subvention. However, In 2005, not only subsidy has decreased but PFU is very low and even if the number will triple until the end of the year, it would not even equal to PFU in 2004. This is due to the financial situation of the population. The high inflation registered this year has a negative impact on the health behavior of the population. Patients have tendency to do self-medication and health is not a priority anymore.

Apart from that, the tariffs unilaterally decided by the MOH cannot cover expenditures, so CHUM decided to charge patients a little more. Yet, it is still too little for make up for the gap between the receipt and the real cost of a given medical act. Affordable tariffs do not always ensure sustainability of the project.

Financial situation (receipt) of CHUM during and after the project (Unit:1000 Ariary)

(Source: CHUM)

There are about 25 entities dealing with CHUM. The tariff applied to patients reimbursed by their employers is much higher and it can be profitable to CHUM to palliate the low tariff for the public. CHUM has a mission to provide universal service to the public and CHUM does not want to tarnish its image in working too closely with private companies. However, there must be a compromise for its financial sustainability.

2) Personnel aspect

The sustainability of the personnel of CHUM is also quite weak. The increased number of patients leads to the increase of workload which decreases the care hours and retired staff are not even replaced. The lack of staff concerns especially paramedics and eventually supporting staff and administration. This is for example the case of the emergency care unit where there are only two nurses and they have to work 24 hours nonstop every two days. In other units, there are 3 nurses who do rotation during the week. CHUM itself is not able to solve that problem, for recruitment of personnel is the responsibility of MOH. Only MOH decides the number of staff members coming to CHUM despite of the need of the latter. CHUM is unable to recruit part time nurses, either.

Sustainability of inputs for medical care:
The equipment is extensively used and the majority is still functioning. 212 equipment items have been installed. 83% are still functioning, and 58% are regularly used. Yet, the prospect for their sustainability is highly questionable. The success of the project increases workload and leads to an overuse of the equipment: they get broken very fast and the very low cost of care makes it impossible to renew them. The maintenance requires a sufficient budget and competent staff. In a financial year, the budget allocated to the operation and maintenance of medical equipment of CHUM as part of the running budget is approximately 5 – 6%. This rate is too low for covering maintenance needs and buying supplies for equipments. Thus, many equipment items may stay dysfunctional for a long time if they have a problem.

3) Sustainability of the project's outputs:

The increased number of the patients:
The number of patients has been increasing during the project and at its termination and has become quite stable later on. But if only outpatients are considered, in 2002, the number was estimated at 10,359 and it decreased to 8,685 in 2003. Even if the number of patients received at CHUM does not increase, the population has confidence in CHUM and the data provided by the service of laboratory and the medical imaging shows that the number of users still increases after the termination of the project.

Number of patients received at CHUM after the termination of the project

Number of analysis and radiography at CHUM

(Source: CHUM)

Increased referral cases:
The number of referral cases increased during the project but then decreased a bit because little has been done after the project. Follow-up at the regional level is not ensured and CHUM managers stressed the need for stronger collaboration with DRS. The referral system and referral cards are not yet extended to private health centers. Due to the lack of budget for field visits, the staff members of CHUM are not able to do field work anymore as they did during the project. The project proved that direct contact was very efficient in earning the trust of the population in rural areas. Sustainability of the referral system depends also on number of staff members, equipment in CSBs and training. If CSBs are strengthened, self-referred patients at CHUM may decrease.

Improved patients' satisfaction with medical services of CHUM:
CHUM has no data on the satisfaction of the patient. Accordingly, a focused group discussion was done in 3 communities by the evaluation team. Most interviewees see that, due good equipment, the cleanness of CHUM and the quality of services have greatly improved. However, they have pointed out some bad habits of CHUM that are still remaining. They find some staff members unkind, the waiting time too long in a non-emergency case, and there are no visits of doctors on weekends. In fact, much of the complaints come from self-referred patients who do not have any understanding of the function of a referral hospital. CHUM makes no distinction between self-referred and referred patients with regard to the waiting time.

Introduction of affordable tariff for the poor:
Normally, the tariffs are affordable for patients but some are still complaining about expensive fees at CHUM. The PFU is a barrier to accessing hospital care, especially for the poorest. MOH has then recently set up the Equity Fund (EF) system to give free medical care to indigents.¹

The EF is not formally installed yet. Normally, a special fund from MOH and a part of medical fees received from patients should be put in a sub-account for equity fund. Such process is under way. CHUM, with an insufficient budget for indigents, has taken some steps to help them. But they are currently not functioning:

- During the project, CHUM set up a working group to care for indigents. The group set identification criteria for real indigents and formulated an indigent card which was different from that in CSBs. But this group is no longer functioning as some members left.

- Overloaded, CHUM is reducing the number of indigents that it takes care of.

EF 2003- 2005 (Unit: Ariary)

CHUM tries to coordinate activities with the other stakeholders. It is proposing to a Catholic congregation to work together on accurately determining the number of indigents. An entity that takes care of patients is the Sisters of Sacré Cœur de Jésus et de Marie. Its prime targets are referred patients from enclave areas and patients of diseases that need long treatment such as tuberculosis. It engages in activities including the following:

- Free medicines with advice from doctors of CHUM (4,800,000 Ariary per year)

- Distribution of food 3 times a week for 60 to 95 persons (13,000,000 Ariary per year)

- Loans for patients who come from very far and have no more resources.

When comparing the expenditures for indigents of CHUM and Sacré Cœur de Jésus et de Marie, CHUM's expenditure amounts to only 26% of that of Sacré Cœur de Jésus et de Marie. Thus it is fair to say that the sustainability of the support to indigents is closely linked to the collaboration with other partners.

Improved administrative capacity of CHUM:
In order to solve the problem of waiting time, the French Cooperation has improved the organization within CHUM, especially the route of patients, and has installed a waiting space for the patients. Everyone has recognized that the door service has greatly improved although it is not perfect yet.
Since the departure of the French expert on hospital management in 2002, the post was vacant and the new expert arrived only in September 2005. He will stay at CHUM for one year to support the Director of CHUM.

The sustainability of the improved administrative of CHUM depends closely on sufficient and efficient administrative staff. Much remains to be done. For example, referral data and patients cards are not processed but just piled up. There may be an inexpensive way to take care of the problem.
Financial management is still weak. CHUM does not know the real cost of a given type of care, so does not know the gap between "ideal receipt" and "actual receipt". If they have this data, it would be much easier for CHUM to set strategy to tackle its financial problems.

Information on CHUM being disseminated to patients and local communities:
CHUM has been well known in the targeted 6 districts during the project because the staff members did field work there. There was no information for the public after the termination of the project except during the 80th anniversary of CHUM in 2004. Self-referred patients coming to CHUM still exist and the number of patients is somehow stable. This shows that the former publicity on CHUM still has some impacts now. In addition, CHUM managers have shown determination on increasing publicity of CHUM. When all reforms are done, the public will be informed through such media as radio and TV.

3-2. Contributing and inhibiting factors

(1) (Factor inhibiting sustainability) The lack of personnel, especially paramedics, is the most important inhibiting factor of the project. It affects not only the quality of care but also the attitude of the staff towards patients.

(2) (Factor promoting sustainability) PFU is the promoting factor of the financial sustainability. Without PFU, the project will not survive. However, a low tariff also poses risks to the sustainability of the project.

(3) (Factor promoting impact) Micro projects such as the improvements of referral system have been promoting the project. In fact, it helped improve the collaboration between DRS, CHUM, partners, city and the health services in Mahajanga. Adding to that impact was the commitment of the partner organizations and the staff of CHUM. Activities of DRS and GTZ are big factors that have promoted and are still promoting the project. Their actions at primary and secondary health services in places such as Marovoay strengthened the referral system.

3-3. Conclusion

After the termination of the project on the Global Improvement of CHUM, the ex-post evaluation shows a good impact on the referral system in Mahajanga despite of some weaknesses of the sustainability of the counter referral system. The impact of the project on students and trainees at CHUM is also positive. The impact would have been totally positive if, during the project implementation, the means to secure the management and maintenance of equipment had been set. With regard to sustainability, financial sustainability is still ensured by the FPU. However, it will be at risk in the near future if additional measures are not taken. Equipment items are also still functioning but a strategy should be found to lengthen their sustainability.

3-4. Recommendations

To CHUM:

(1) The problem of maintenance may be tackled by outsourcing some of the functions of the O&M department. This may be able to reduce the O&M cost of the CHUM and efficiently maintain the equipment in the hospital.

(2) CHUM needs to tackle the problem of the shortage of paramedics. One possibility is to give inactive nurses in the area or recently graduated nurses the possibility of practicing their knowledge at CHUM as interns or at low cost with flexible working hours.

(3) If the tariff of each medical care is calculated and known, it will be easier for CHUM to see the gap between the receipt and the real cost of a given medical act. With this data, it is easier to convince MOH and partners on how things should be done. CHUM will request partners to give this task a priority.

(4) When CHUM does its budgeting, it can use some part of the budget for publicity efforts through radio, TV, posters, and pamphlets. For the population in rural areas, fieldwork of the staff members of CHUM is a good way to promote RS and the tariff for the public. That will also enhance the image of CHUM to the communities at District Hospital (CHD) and CSB level.

To MOH

(1) The MOH can support CHUM much more if it recruits staff members, especially paramedics, not only for the sustainability of the project but also because the working conditions of the personnel affect the care provided to the patients. MOH can also send recently graduated nurses to work at the hospitals as trainees. In that way, the cost would be lower than recruiting new staff members.

(2) The department responsible for maintenance at the central level can study the possibility of standardization of the types of medical equipment utilized in Madagascar according to the economic situation and the capacity of the technicians. Such measure would also make it easier for the provider to run and keep spare parts and supplies to users for a long period of time.

(3) Setting a price including maintenance cost and taking into account the purchasing power of the population would be the best solution to fix the PFU. But for public hospitals, the price is set by MOH. Subsidy from the government would be a solution to fill the gap between low tariff and real value of a given act of care. It will also allow the health center to maintain its equipment, and undertake activities to improve itself and services it provides.

To JICA

(1) The availability of spare parts after the guaranty period is seen as a major problem in keeping equipment functional. In some cases, equipment cannot be used due to the unavailability of spare parts (e.g., main cards for the Spectrophotometer) although equipment itself is functional. JICA may be able to help solve this problem by communicating with the manufacturers and distributors in Japan.

(2) JICA can help CHUM improve its sustainability by sending an expert to CHUM to help with calculation of the tariff and data processing.

¹ Indigents are the poorest people constituting about 10% of the population