Ex-post Evaluation

1-1 Background and Overview of the Cooperation

The petroleum industry plays a major role in the United Mexican States (hereinafter referred to as Mexico). Throughout the 1990s, the revenues of the Mexico Petroleum Public Corporation (Petroleos Mexicanos: PEMEX), a government-owned corporation, reached nearly 30% of public revenues and 7% of GDP, contributing much to the national economy. However, the PEMEX oil refineries were more accident-prone than those in developed countries. Training to prevent accidents did not accomplish much. Improvement of safety in the PEMEX refineries was thus a matter of high national importance and an issue for the Mexican economy as a whole.

Under such circumstances, in 1995 the government of Mexico requested the government of Japan for comprehensive cooperation on safety and maintenance technologies including everyday maintenance. Consequently, the Japanese government carried out a pre-project study in April 1996 and a long-term study in July 1996, respectively, then signed and exchanged R/D in September 1996. Then, for five years from December 1996 to 30 November 2001, the Japanese government implemented a project-type technical cooperation for the purpose of improving the safety of the Salamanca oil refinery (Refineria Ing. Antonio M. Amor, hereinafter referred to as RIAMA).

1-2 Description of the Cooperation

This project carried out the following:

• Establishment of the Training Center for Oil Refinery Safety (Centro de Entrenamiento en Seguridad, hereinafter referred to as CES);
• In the first half of the cooperation period, activities centered on development of a training course on safety directed to employees of the Salamanca oil refinery;
• In the latter half of the cooperation period, in addition to the implementation of the training, on-site guidance to promote safety activities at the refinery, and review of organization, systems and regulations.

(1) Overall Goal

The productivity of the Salamanca oil refinery improves.

(2) Project Goal

The safety of the Salamanca oil refinery improves.

(3) Outputs

1) The organization and operation systems of the project are established.

2) All employees acquire knowledge on safety.

3) Employees' attitudes toward work improve.

4) All employees foresee potential danger at work and take preventive measures.

5) All employees comply with the proper procedures and the regulations.

6) Awareness of unsafe situations increases.

7) All sections and divisions utilize information on safety.

8) All sections and divisions implement safety activity plans.

(4) Inputs

Japanese side:

Mexican side:

3-1 Confirmation of Accomplishments

(1) Impact

The Overall Goal of this project is "The productivity of the Salamanca oil refinery improves", and its indicator is "Unplanned plant closures due to human error decrease". To measure impacts, the Project Goal and the Overall Goal must be logically connected, and the extent of achievement must be measurable. The existing PDM does not indicate sufficient consideration on logic at the time of PDM formulation. Moreover, it was hard to obtain specific data during the implementation of the ex-post evaluation. Thus the evaluation team was unable to measure in detail the extent of achievement of the Overall Goal. Given these constraints, the evaluation team did its best to measure the Overall Goal with available data.

The number of plant closures due to human error decreased significantly in the late period of the project implementation, but increased after the end of the project. The same trend applies in the ratio of plant closures due to human error in all plant closures. These results point to the conclusion that safety was promoted during the project period but the effect of the promotion did not last after the end of the project.

According to the comparison of the planned quantity of crude oil refining and the actual quantity, after the project period, the refinery reached the planned quantity almost every year until 2004. However, it is hard to measure the extent of the achievement of the Overall Goal with this indicator alone because RIAMA changed the planned quantity frequently.

According to the numbers, there is no correlation between the number of plant closures and the ratio of actual crude oil refining. This result suggests either of the following: (1) RIAMA changed the planned quantity of crude oil refining in accordance with the decrease in production caused by plant closures; (2) There is no correlation between the number of plant closures and the quantity of crude oil refining. However, the evaluation team was unable to obtain relevant information and find out why no correlation was seen. Thus the evaluation team was unable to confirm "improvement in the productivity of RIAMA from fewer plant closures due to human error".

At the time of the terminal evaluation, the evaluation team saw the following impacts: (1) The RIAMA employees have better awareness of safety, resulting in an improved work environment, better attitudes toward work, and decrease in the number of accidents; (2) CES became the strategic focal point for disseminating safety methods in the entire PEMEX oil refinery system; and (3) Safety methods were disseminated to project areas and surrounding regions. However, (3) was the only impact that was maintained at the time of the ex-post evaluation.

(2) Sustainability

The evaluation team believes that technical sustainability of the project has been ensured as the counterparts have developed technologies introduced during the project implementation and understand the significance of the introduced technologies although some of the technologies have been slow to sink in. The organizational structure has not been stable as it has seen a major change since the end of the project. On the financial side, the budget has decreased considerably, calling into question its sustainability.

1) Physical and Technical Sustainability

[Technical level of the counterparts] Since the end of the project, the Plan for Training and Advice of CES has been established and carried out in five PEMEX oil refineries. The plan is to study and diagnose the management situation and organizational structure on safety at each of the refineries and implement appropriate training and advice based on the diagnosis. It is a method that the counterparts developed on the basis of the method and techniques that experts introduced during the project period to diagnose the management situation and organizational structure on safety in any project. In addition, since the end of the project, the counterparts have developed "Cursos Curzados", a self-education system for on-site refinery workers that can evaluate their learning process on the Internet. This system was introduced in a part of RIAMA in 2004, will be adapted in RIAMA as a whole in 2005, and will be introduced in other refineries as well.

[State of the retention of the safety management methods that the project transferred to the counterparts] Since the end of the project, of all the safety management methods, the retention rates of 5S, KYK, and the safety activity plan are decreasing in particular. However, a questionnaire survey in this evaluation shows that 86% of the refinery section chiefs feel that attitudes of their subordinates at work have been improving. The counterparts have been implementing the training by CES even after the end of the project while revising and formulating educational materials. They have also put together a brochure on the training courses to disseminate CES activities outside RIAMA. According to the counterparts, they focused on training outside RIAMA until 2003 but have been putting priority on training inside RIAMA since the current director took the helm in 2004.

[Evaluation of the project from the stakeholders] The evaluation team asked the RIAMA area chiefs and sections chiefs to evaluate the training by CES. Their responses were mostly favorable. To the question on whether the RIAMA leadership has changed its attitude toward safety management, more than half of the chiefs said that they had seen some change, but less than half of the chiefs thought that the leadership had taken an interest in safety management activities and provided support. The evaluation team also carried out a questionnaire survey on five oil refineries in Mexico other than RIAMA, and received replies from four of them. The survey indicated that the four refineries understood CES activities and were disseminating and implementing the Japanese-style safety management methods. One of the four just started introducing the methods.

[State of the maintenance and usage of the equipment provided] Most pieces of the safety text equipment are generally in good condition. However, some pieces of the office equipment and the training materials are out of order, and only a fraction of the technicians use the English versions of the written materials. The video tapes are used in training sessions.

2) Organizational Sustainability

[Organizational System of RIAMA on Safety Management] RIAMA has the Safety and Environment Division that is responsible for the safety training set up by the 5S secretariat and the project as well as for safety management of RIAMA itself. During the project period, 13 counterparts were deployed at CES. The PEMEX headquarters gave them a temporary status as CES staff members and paid them. However, PEMEX abolished these posts in 2003 due to budget cuts, and the counterparts returned to their former posts or got transferred elsewhere. 8 counterparts were reassigned to CES to continue CES activities after the end of the project period, but the number is currently down to five as a few resigned. Other staff members at CES, such as clerical staff, secretaries, and drivers, have also left. Moreover, as part of corporate downsizing, PEMEX has also abolished the special promoter system set up during the project period. Most of the 20 promoters returned to their erstwhile posts and only one of them is currently assigned to CES as an instructor.

[Standing of CES in PEMEX] According to the PEMEX Safety Director, CES is responsible for implementing safety improvement activities in 6 PEMEX oil refineries and other PEMEX agencies, supervising staff members of refineries other than RIAMA who have undergone the CES training, and disseminating safety management methods in those refineries. However, in an interview by the evaluation team, the counterparts stated that the training for other refineries was not done at their request but at the initiative of CES or a message from PEMEX, and CES was not particularly well known. In addition, they said, the current RIAMA director who took up the post in 2004 instructed CES to put priority on activities for improving safety at RIAMA rather than at other refineries, and the number of training sessions for other refineries is on a downward trend. Out of the 10 counterparts who consented to the interview, 7 think that the standing of CES in PEMEX and RIAMA has not been ensured because CES has no permanent staff members and receives no funding from PEMEX.

3) Financial Sustainability

After the project, PEMEX provided financial aid to CES until FY 2002 but cut it off in FY 2003. Currently, the personnel cost of the staff members seconded to CES is paid by the Safety and Environment Division and the Human Resources and Training Section of PEMEX. The Human Resources and Training Section also pays for any extra expense incurred in implementing CES activities. In 2004 and 2005, the Human Resources and Training Section included the personnel cost and miscellaneous expenses of CES in its budget, but the amounts are not sufficient.

3-2 Factors That Contributed to the Achievement of Outputs

(1) Project Plan

• The counterparts at CES are highly skilled and interested in safety management.

(2) Implementation Process

• Since a portion of oil refinery areas and work areas has retained safety management methods, employees nearby have experienced for themselves the effects of the methods.

3-3 Problems and Their Causes

(1) Implementation Process

• CES did not secure for itself a strategic position in the organizations of PEMEX and RIAMA.
• CES has neither a budget nor permanent staff members.
• Turnover in the leadership ended up weakening safety activities.

3-4 Conclusion

The accident rate at RIAMA has been slightly increasing since the end of the project. The number of plant closures due to human error was on a downward trend in the first half of 2004 but was on the increase from the latter half of the year up to the time of the ex-post evaluation. Thus it is not fair to say that safety has improved. On the technical side, the CES counterparts have been improving the tranferred technologies and continuing the safety management training. They are motivated to create an organization capable of diagnosis and advice. Accordingly, technical sustainability is high. However, organizational sustainability is low because CES has lost a clear standing not only in PEMEX but also in RIAMA and has secured neither sufficient human resources nor budget for continuous activities. The future of the safety management activities in PEMEX and RIAMA rests on whether the leadership raises its awareness of safety management and strengthens institutional and financial support to the activities.

3-5 Recommendations

To disseminate further the technologies developed by the project and make larger impacts, the evaluation team recommends the following:

(1) PEMEX should have a clear strategy on safety management and utilize the project outputs accordingly.

(2) PEMEX should clarify the standing and functions of CES in the organization of its headquarters. In addition, PEMEX should assign permanent, full-time staff members to CES to keep CES going.

(3) To achieve (1) and (2), the PEMEX leadership is to undergo training on safety management and recognize its importance.

(4) To solve in a fundamental way the problems that this evaluation found such as the PEMEX leadership's lack of support to safety management and lack of information disclosure at PEMEX, and ultimately for the sake of Mexico's economic, social and political development, PEMEX as a whole must acquire the perspective of Corporate Social Responsibility (CSR).

3-6 Lessons Learned

(1) To secure the impact and sustainability of the project, the project team should formulate approaches and a project design that take into consideration the process toward achieving the Overall Goal.

(2) To manage the project in a secure and effective fashion, the project team should set practical and measureable indicators.

3-7 Follow-Up Situation

After the end of the project, a senior volunteer was dispatched to CES in April 2002 and followed up on the project for two years.