Ex-post Evaluation

1-1. Background of the Project

Health indicators of Jamaica are at relatively good levels. For example, the child mortality rate for males and females was 21 and 19 per 1,000 respectively, and the life expectancy at birth for males, females, and the total population was 71.0, 74.0, and 73.0 years respectively in 2003. However, chronic lifestyle diseases (CLD's), such as hypertension and diabetes, have been increasing along with negative lifestyle changes and the aging society. The difference in health care services between the urban areas where around 52% of its population is concentrated and other areas was the major issue in Jamaica.

Under these circumstances, the project was initiated in the southern region (three pilot parishes of Manchester, St. Elizabeth, and Clarendon) whose health care was substandard compared to other areas, in order to improve the health of the people in this region. The aim was to enhance the medical health care system with a specific focus on education in health care related to CLD's and the prevention of diseases.

1-2. Project overview

To enhance the regional health systems in Jamaica, the project implemented activities for the health care workers through cooperative activities, such as the organization of disease prevention programs, health examination, counseling activities, textbooks on health care education, and health awareness in order to prevent CLD's.

(1) Overall Goal

The health status of the population of Jamaica is improved by strengthening the function of the regional health systems.

(2) Project Purpose

The health care system in the southern region is strengthened, focusing on the prevention of chronic lifestyle diseases (CLD's).

(3) Outputs

a) The administrative/organizational capacity of the Southern Regional Health Authorities is improved.

b) The functions of parish health center facilities are improved.

c) Human resource skills are improved.

d) A CLD prevention model is developed and implemented in the pilot parish, Manchester.

e) The CLD prevention model is extended to St. Elizabeth and Clarendon.

(4) Inputs (as of the Project's termination)

Japanese side:

Jamaican side:

3-1 Summary of Evaluation Results

(1) Impact

a) Achievement of the Overall Goal

The indicator of the overall goal is described as the "number of sustainable wellness activities in the regions." If "promotion of health examinations" which is one of components of wellness activities is expressed in a numerical value, it seems that the "number of patients who visit health centers, including fixed and mobile clinics, for having health examinations" is appropriate. Because the system in which the patients are only able to have health examinations has not been established other than in the southern region (target area), the "number of patients" in two target parishes is shown in the below table. The total number of patients decreased in Manchester in 2004 and increased in St. Elizabeth for three years. It is considered that health centers in Manchester were directly affected by the termination of the project since the SRHA is in Manchester. Also, more patients visit the fixed clinic to have health examinations in Manchester, compared to the mobile clinic. In St. Elizabeth, on the other hand, patients are more willing to do health examinations if the mobile clinic travels to their communities as shown in the below Table.

According to the questionnaire survey on the frequency of exercise to the patients in three parishes by the Study Team, 42.2% of 90 respondents replied that they had only started exercising after they had received health examinations or health education. Also, more than 87.7% of the respondents have increased the number of times they exercise each week. This result could be highly evaluated as a movement for the improvement of the health status.

The wellness activities for persons in the community, including "promotion of health examinations" for the patients with or without diseases and "extension of health education," have actively been implemented in other regions. However, it cannot be concluded that this project alone impacts the wellness activities in other regions because of the existence of other national programs, such as National Strategic Plan for the Promotion of Healthy Lifestyles in Jamaica (2004-2008), being carried out at the same time. Further, in order to replicate the same model used by the SRHA in other regions, it may be more appropriate for using the concepts and principles of the model in other regions since it is necessary to prepare medical equipment and facilities utilized by this project. However, the SRHA does not document the process, experiences, and outcomes of the project so as to apply the SRHA wellness model to other regional health authorities, so it is necessary for the SRHA to prepare the manual for them.

As a follow-up of this project, the Third Country Training Program has been launched for five-year period. This Program might be a significant medium for extending the concepts and principles of this project because participants in the Program were representatives from not only other Caribbean countries but also three other regions in Jamaica. Moreover, parts of this project are shared with other regional health authorities through the quarterly National Review meetings, etc.

Specific indicators of the overall goal were not clarified at the start of the project or during the project. It is difficult to measure the attainment of the overall goals of the project without setting up these indicators. Appropriate indicators are necessary for grasping the contents of the overall goal and monitoring the project activities.

b) Positive or negative impacts

The SRHA wellness model has influenced a new proposal for "A Wellness Model to Chronic Non-Communicable Diseases: Prevention and Control", which was prepared and sent by the West Regional Health Authority (WRHA) to the National Health Fund (NHF), in order to request the financial support. The proposal was spearheaded by the WRHA participant in the Third Country Training Program. The orientation of this proposal is quite similar to this project.

According to the questionnaire survey in three parishes by the Study Team, 94.9% of 137 respondents answered that they had ever heard about CLD's through the TV, radio, newspapers, church, community meetings, friends and relatives, and so forth. Of those respondents, 92.0% of 125 patients selected either "5: strongly agree" or "4: agree" with their interests in having health examinations. Moreover, 97.4% of 116 patients responded that they selected either "5: strongly agree" or "4: agree" with their interest in learning more about CLD's through health education. Therefore, the PR activities encourage the patients to have health examinations and learn more about CLD's through health education.

In terms of communication with relevant organizations of this project, the vision and the concepts of the project were not shared in order to collaborate with each other. This is because the Heart foundation and other organizations already offered similar services regarding CLD's in the areas that the project was targeting. The lack of communication between the SRHA and the Heart Foundation of Jamaica especially led to competition over the offering of the service rather than collaboration. As a result, the Heart Foundation stopped doing annual screenings at the health centers in the southern region.

(2) Sustainability

a) Policy aspects

In order to address the problems of CLD's, the MOH has put in place a national strategic plan titled "The National Strategic Plan for the Promotion of Healthy Lifestyles in Jamaica 2004-2008". The Plan is spearheaded by the Division of Health Promotion and Protection in the MOH and is being developed and implemented in collaboration with other agencies of government, the private sector, NGO's, and other international organizations, including PAHO/WHO, UNICEF, and USAID. The aim of the Plan is to promote "healthy lifestyles" in the population, so as to reduce the risk of developing heart disease, diabetes, hypertension, obesity, cervical cancer, and HIV/AIDS as well as to reduce the incidence of violence and injuries.

Furthermore, the SRHA has the Strategic Development Plan 2002-2007, which outlines strategies partially to address CLD's. The SRHA has focused on the prevention activities so as to promote wellness activities against CLD's through heath education and counseling for the patients.

b) Administrative and financial aspects

In the southern region, the preventive activities against CLD's are monitored by the steering committee called JACOSH (The Jamaica-Japan Cooperation on Strengthening Health Care). Specifically, JACOSH monitors the activities at the fixed and mobile clinics. Furthermore, it manages not only the follow-up activities of this project but also activities related to the National Healthy Lifestyles program which not only addresses CLD's but also reproductive health as well as injury and violence.

The JACOSH members include the Regional Director and Regional Technical Director of the SRHA, Parish Managers, Health Education Officers, and health care workers from the three parishes, and they meet every two months.

As financial aspects, the fees collected by the clinics are put directly into the general accounts of the financial division at the SRHA and then disbursed to the health centers according to their needs because the MOH has recommended that specific accounts should not be kept for particular purposes. According to the Regional Technical Director of the SRHA, the clinics might obtain more funds from the SRHA than the fees they hand over to the SRHA financial division from the patient fees. This is because the patient fees collected at fixed and mobile clinics are lower than the market prices. Further, the main financial requests from the fixed and mobile clinics are for equipment and maintenance, which cost a lot. In terms of the cost-effectiveness of the SRHA wellness model, therefore, it might be difficult for other regions to replicate the model because of budgetary constraints. Thus, since the wellness activities for preventing CLD's are expensive, it would be necessary for the SRHA to secure funds for the wellness activities from the NHF which emphasizes health promotion and illness prevention, etc.

According to the questionnaire survey in terms of how patients perceive the fees for health examinations, 73.3%, 8.9%, and 4.4% of 135 respondents replied that the fees for the health examinations are "appropriate", "cheap", and "very cheap" respectively.

c) Technological aspects

According to the questionnaire survey to the 140 patients in three parishes by the Study Team, 96.2% of 130 respondents answered that the health care workers and community health aids (CHA's) were either "5: very capable" or "4: capable" enough to promote the preventive activities against CLD's.

In terms of biomedical equipment, there are only two technicians for biomedical equipment in the southern region who must take care of 85 health centers, 5 hospitals, and a community hospital. They mainly carry out both preventive and corrective checks. Preventive checks are conducted on equipment on a quarterly basis. Therefore, the number of capable technicians is not sufficient.

3-2. Factors that have promoted project

(1) Sustainability

Although the purpose of the Third Country Training Program is to provide the participants from selected CARICOM member countries with an opportunity to improve their knowledge and techniques in the field of the prevention of CLD's, representatives from three regions in Jamaica have also attended this Program. The participants of the Program visit the model sites in the southern region, in order to absorb the knowledge and techniques on wellness activities at the fixed and mobile clinics. Because the health care workers in the SRHA are observed by the participants from other CARICOM countries and other regions in Jamaica, it is expected that they might continuously make efforts for brushing up their knowledge and techniques in order to act as role models for preventing CLD's in the CARICOM regions. Furthermore, as the WRHA participant attended the Third Country Training Program, the WRHA tries to apply the SRHA wellness model to the western region by preparing the proposal and sending it to the NHF. Thus, it is expected that other regional health authorities have sustained their efforts continuously.

"The National Strategic Plan for the Promotion of Healthy Lifestyles in Jamaica 2004-2008" by the MOH promotes "healthy lifestyles" in the population of Jamaica, so as to reduce the risk of developing heart disease, diabetes, hypertension, obesity, etc. In the policy aspects, therefore, the environment in which the project activities related to CLD's have continuously been promoted is established.

The steering committee called JACOSH manages regular meetings every two months and the follow-up activities of this project even after the termination of the project. Thus, JACOSH performs the role for promoting the activities continuously through the monitoring activities, etc.

3-3. Factor that have inhibited project

(1) Impact

Indicators of the overall goal were not set up appropriately. Although the indicator of the overall goal described as the "number of sustainable wellness activities in the regions" is established, it is difficult to determine whether or not the overall goal has been achieved. Without setting up the specific indicators, counterparts are not able to confirm the attainment of the overall goal.

One of obstacles is the question of who, in the other regions, would have the responsibility for replicating the SRHA wellness model. The most appropriate personnel are the technical directors in other three regions with the assistance of JACOSH and the CD Unit (Chronic Diseases Unit) under the Health Promotion and Protection Division of the MOH.

While health education for health-seeking behaviors might influence many young people and adults, they are coming in touch with visual messages daily through television advertising and being enticed by North American fast food advertisements. The aim of the health promotion activities is to promote healthy lifestyles including "diet", so the aim is contrary to the contents of the fast food advertisements. Therefore, the messages sent by the fast food advertisements are one of factors inhibiting the attainment of the overall goal.

From the perspective of diet, two factors inhibiting the achievement of the overall goal were noted in the interview survey. First, most of the population are not aware of the importance of nutrition labels on food items. This lack of awareness has been due to low literacy levels. The main focus of patients is not the quality of food eaten but the quantity, in terms of healthful eating. Secondly, in poor areas of towns and cities, it is very difficult for the people to buy healthy foods, such as fruits and vegetables, because they are more expensive than high fat foods which are cheaper to buy. The availability of the healthy foods has become lower due to droughts and hurricanes. As a result, they eat less of expensive and healthy carbohydrates, such as yams and bananas, and consumed more affordable carbohydrates, such as flour. Consequently, the poor people buy and eat more high fat foods than low fat foods due to low literacy levels and economic reasons.

(2) Sustainability

It was difficult for the Study Team to collect data in the southern region. In St. Elizabeth, for example, a statistician has not been deployed for the purpose of the data analysis. However, a community nurse manually collected the information to complete a data sheet without computerized data. Additionally, the community nurse in St. Elizabeth could provide the data sheet for the Team earlier than statisticians in Manchester. Also, the data sheet was not provided for the Team from Clarendon. It might provide an indication of what is going on after the termination of the project in terms of data collection and analysis on CLD's in the southern region. Thus, unless skilful statisticians are deployed for data analysis, project activities, such as "data collection of CLD's" and "consolidation of health statistics", cannot be carried out continuously.

Because the fixed and mobile clinics can obtain more funds from the SRHA than the fees they hand over to the SRHA financial division from the patient fees, it might be difficult for other regions to replicate the SRHA wellness model in terms of the cost-effectiveness of the model. Therefore, the wellness activities are expensive and the budget is limited in other regional health authorities, so that it is considered that the sustainability of the project is low in the financial aspect.

3-4. Conclusions

Without setting up the specific indicators of the overall goal, it is difficult to measure the attainment of the overall goal. Hence, it is desirable to establish the appropriate indicators so as to grasp the contents of the overall goal and to monitor the wellness activities.

Further, in order to replicate the same model used by the SRHA in other regions, it may be more appropriate for using the concepts and principles of the model in other regions unless other regional health authorities are able to prepare medical equipment and facilities utilized by this project. Therefore, it is necessary for the SRHA to prepare the manual for the purpose of the application of the model to other regional health authorities. Furthermore, the concepts and principles of this project are partially shared with other regional health authorities through the Third Country Training Program and the quarterly National Review meetings. Overall, it is considered that the impact of the project is low at this stage, but it is expected that the overall goal has been achieved earlier by the efforts of the counterparts.

From the perspective of the project sustainability, there are advantages and disadvantages as mentioned below. In the policy aspects, the environment in which wellness activities have continuously been promoted is established by "The National Strategic Plan for the Promotion of Healthy Lifestyles in Jamaica 2004-2008". Also, JACOSH manages regular meetings every two months and the wellness activities after the termination of the project. Moreover, as the WRHA participant attended the Third Country Training Program, other regional health authorities try to apply the SRHA wellness model to their regions. These advantages contribute to continuous implementation of wellness activities.

On the other hand, data collection and analysis on CLD's were not conducted with appropriate database in the southern region, and this fact indicates that the sustainability of the project activities is low. In terms of the cost-effectiveness of the model, the fixed and mobile clinics can obtain more funds from the SRHA than the fees they hand over to the SRHA financial division from the patient fees, so that the sustainability is low in the financial aspect, too. Altogether, it seems that the sustainability of the project is not high so far, but it would be possible to enhance the sustainability of the project by strengthening the collaboration with the NHF, etc.

3-5. Recommendations

To the SRHA and MOH:

As mentioned earlier, there is the question of who will be responsible for replicating the SRHA wellness model in the other regions. Thus, the mechanism in collaboration with the JACOSH and the CD Unit should be established for the achievement of the overall goal as a management body to encourage cooperation among the technical directors in four regions of Jamaica.

The SRHA and the MOH scrutinize the types of health indicators to measure the achievement of the overall goal of the project. For instance, the change in the BMI (Body Mass Index) level of the patients could be an appropriate indicator in order to recognize the attainment of the overall goal and to monitor wellness activities. If the proportion of persons with high BMI levels has fallen over the years, then it might be an indication that the health status of the patients has improved.

Because the fixed and mobile clinics can obtain more funds from the SRHA than the fees they hand over to the SRHA financial division from the patient fees, it is severe to continue wellness activities under this condition in terms of budgetary aspect. Therefore, it is recommended that the SRHA enthusiastically prepare the proposal and submit it to the NHF which contributes to the prevention of CLD's, in order to secure funds for the wellness activities.

It is very crucial to use the results of the data analyzed by the statisticians in each parish because the results should be fed back into the development of regional plans in the southern region. To this end, exclusive statisticians should be deployed in the three parishes and collaborate within the newly established working group, consisting of the statisticians from three parishes.

The SRHA should prepare a manual containing the concepts and principles of the SRHA wellness model including the method of data analysis, administrative techniques, etc., in order to share pieces of the model with other regional authorities.

3-6. Lessons learned

It will be extremely difficult to track the attainment of the overall goal of the project unless the indicators are precisely set up. Thus, the key indicators should be clarified before or during the cooperation period. At the same time, the means of verifications must be examined thoroughly in terms of cost, the credibility of data sources, and ease/difficulty of obtaining data. The Ministry concerned should confirm key indicators in the ex-ante or mid-term evaluation study through extensive involvement.

In order to attain the overall goal, it is necessary for the project to have a clear vision of how the project purpose can lead to the overall goal under the instruction of the JICA project team. Before the termination of the project, therefore, the mechanism through which counterparts are able to follow the procedures for extending the project model to other areas should be established.

In order to extend the concepts and principles of the project model from the target area to other areas, the process, experiences, and outcomes of the project should be documented so as to assist in replicating them at the relevant institutions in the other areas.

An external workshop for the relevant organizations or groups should be held before the launch of a newly established project in order to share the vision and the concepts of the project with them. Also, the planned counterparts of the project, through an internal workshop, should also have the opportunity to discuss and analyze the stakeholders of the project through the stakeholders analysis of the PCM workshop, especially beneficiaries and negatively affected groups of the project, before launching the project.

3-7. Follow-up Situation

As follow-up activities, the Third Country Training Program has been conducted once a year from Japanese fiscal year (JFY) 2003 to JFY 2007. The first and second year programs were hosted by the SRHA. On behalf of the SRHA, the MOH was to be responsible for the implementation of the Program in JFY 2005 in the third year program, but it was cancelled because of a lack of management by the MOH. At the present moment, the schedules of the fourth and final year programs are undecided.