Ex-post Evaluation

1.1 Project Background:

In June 1990 the Collor administration (March 1990 - October 1992) implemented the Brazilian Program of Quality and Productivity (PBQP), seeking to improve the quality and productivity of the national industry by promoting deregulation, liberalization and the introduction of foreign capital, besides the privatization of public companies. Numerous organizations participated in the program, with projects being developed in each of the 16 participant States.
The Brazilian Government next planned to establish Brazilian Institutes of Quality and Productivity (IBQP) in five States, requesting technical cooperation from Japan within this framework.
In response to this request, the Japanese Government supported the said program by implementing the project during the period of 06.01.1995 to 05.31.2000.

1.2 Project Overview:

The goal of this study was to analyze the performance of IBQP-PR (the Institute of Quality and Productivity in Paraná) considering "impact" and "sustainability".

(1) Overall objective:

Dissemination to Brazilian society of the IBQP-PR concept and technology to improve productivity.

(2) Project objective:

Qualification of IBQP-PR concerning technology improvement and development and awareness of increased productivity.

(3) Expected results:

• Established administration system for the project.
• Equipment needed for the implementation of the activities in a perfect state of conservation and operation.
• Technical capacity of the counterparts in their respective improved areas.
• Implemented and monitored seminars and trainings.
• Systematically implemented consultancy services.
• Systematically implemented public relations services and promotion of increased productivity.

(4) Investments:

By Japan:

By Brazil

3-1 Summary of evaluation results:

(1) Recognition of the activities developed

a) Overall Objective- (Dissemination to Brazilian society of the IBQP-PR concept and technology to improve productivity): reached, in spite of being restricted to Paraná and Santa Catarina in terms of indicators (expansion of IBQP-PR activities its network and increase in the number of beneficiaries' of the IBQP-PR section, social recognition of IBQP-PR).

b) Project Objective (Qualification of IBQP-PR concerning technology improvement and development and awareness of increased productivity): fully reached, as indicated by the numbers obtained in the evaluation study regarding the PDM indicators (an increased number of sectors and beneficiaries of IBQP-PR, degree of satisfaction of the beneficiaries of IBQP-PR activities).

(2). Impact:

Regarding the two basic questions proposed in this project, the first one sought to know if concerning the "impact", the projects developed by IBQP-PR reached the expected results. Data analysis showed that 60.71% of the companies confirmed their degree of satisfaction while only 17.86% showed dissatisfaction.
The conclusion is that IBQP-PR obtained positive results regarding impact.

(3) Sustainability:

Regarding "sustainability" the study sought to establish whether the projects developed by IBQP-PR reached the expected results. The study revealed that 35.71% of the companies studied confirmed their degree of satisfaction while only 7.14% confirmed their degree of dissatisfaction.
The conclusion is that IBQP-PR obtained satisfactory results regarding sustainability.

3-2 Factors that promoted sustainability and impact

The factors listed below doubtless helped reach the objectives proposed in the PDM plan. During the study the clients expressed favorable opinions concerning the services rendered by IBQP-PR. These were registered in percentiles.

Regarding impact, the factors that promoted success were the following:

• "effectiveness" (78.57%) – "it sought to establish the opinion of the respondents on the effectiveness of the programs developed and/or implemented by IBQP-PR within the organization ";
• "applicability" (57.14%) – "concerning the actual applicability of the programs developed and/or implemented by IBQP-PR in the organization";
• "expectation" (71.43%) – "whether the work carried out by IBQP-PR met the initial expectations of the organization ";
• "technical capacity" (64.29%) – regarding the technical capacity of the IBQP-PR staff in proposing solutions to meet the needs of the organization ";
• "performance" (71.43%) – whether there was a significant performance improvement in the area where IBQP-PR rendered services ";
• "degree of satisfaction (60.72%) – it sought to analyze the general results of the variables that measured impact.

We conclude that regarding impact, IBQP-PR had good results. However it is important to note that in the "positive effects" variable ("whether or not there were unexpected positive effects throughout the programs implemented by IBQP-PR in this organization") the results were unsatisfactory. Even though there was a good acceptance by the clients, the conclusion is that there were no unexpected positive effects from the activities.
The sustainability of IBQP-PR actions is proven by the 71.43% growth in the number of clients between the years of 2002 and 2003, with emphasis on the following criteria:

• "communication and public relations" (85,71%) – "regarding the grade that can be attributed to the work relations between this organization and the Brazilian Institute of Quality and Productivity (IBQP-PR) ";
• "punctuality" (85,71%) – "whether or not IBQP-PR met the deadlines established for the programs developed and/or implemented ";
• "costs" (64,29%) – "regarding the costs presented by IBQP-PR in exchange for the services rendered ";
• "technical materials" (61,54%) – "regarding the quality of the technical materials used by the IBQP-PR team in the implementation/development of the programs (when fitting) ".

These percentages make it possible to affirm that the number of clients who declared being satisfied with the activities developed by IBQP increased.

3-3 Factors inhibiting sustainability and impact

The study indicated that no factors were found to inhibit the impact of IBQP activities, however, as detailed in item 3.3, in the variable "positive effects" the degree of satisfaction was of 21.43%, the same as the degree of dissatisfaction, while 57,14% preferred not to give an opinion. The conclusion therefore is that although there were no inhibiting factors, there were no significant positive impacts either.
According to the clients among the factors that inhibited the sustainability of the services rendered by IBQP-PR those that presented the worst results were:

• "Evaluation" – ("On the systemic evaluation methods of the results of the programs developed and/or implemented by the IBQP-PR in this organization, when fitting" – 21.43%) – ("On the effects verified after the development of the programs by IBQP-PR, in this organization" – 35.71%).
• "Acceptance" – ("Regarding the acceptance of the programs developed and/or implemented by IBQP-PR by the organization's affected employees"). Only 35.71% of respondents confirmed this variable;
• "Communication" – ("Regarding the clarity and effectiveness of the reports presented by IBQP-PR at the end of the services rendered to this organization"). 46.15% of the customers declared to be satisfied with this variable.

The variables show that IBQP-PR needs to implement better monitoring system for its activities as well as improve communication. In spite of being satisfied with the activities carried out by IBQP-PR, the clients did not attribute any positive results to the activity.

3-4 Conclusion

From the point of view of client satisfaction the study revealed a degree of satisfaction confirmed by 50.24% of the sample surveyed against only 16.51% of declared dissatisfaction in the same group.
The study furthermore showed that, in terms of quality, the services rendered by IBQP-PR have a good acceptance among the companies surveyed. However, the greatest problem that compromises the quality of the services rendered by IBQP-PR lies in the "evaluation" component.
IBQP-PR seems not to possess a methodology to evaluate its own work and provide the necessary feedback to allow it to correct small dysfunctions in future work. IBPQ furthermore needs to consider that even work that meets with considerable success in an organization today will not necessary be equally successful in the future. The same company may be going through a period in which the organizational climate is less favorable than during the previous works, something that is not unlikely considering the constant fluctuations of the economy as well as the lack of consistency and high labor turnover in the private sector.
Besides, the lack of a self-evaluation methodology makes it impossible or at least more difficult for IBQP-PR to make significant progress between one project and another.
As mentioned in items 3.3, 3.4, the General View of the Project and the Expected Results, the study led us to conclude that:

a) The overall objective was achieved in terms of concept and technology to improve productivity, but the results remained regionally limited to the States of Paraná and Santa Catarina as a result of the planned gradual expansion to the rest of the country. It is important to note that some of the companies surveyed were not aware of the Institute's work, due to the lack of supervision, monitoring or contact following the IBQP activity. We can therefore conclude that the overall objective was only partially achieved.

b) As to the project objective, IBPQ-PR obtained satisfactory results regarding the improvement and development of quality and productivity.

c) Regarding the expected results:

• As to the administration systems of the established project, the study showed that IBQP-PR had satisfactory results. This is because sustainability and impact were considered to be part of the administration system.
• As to the equipment needed for the implementation of the activities, part of it lost its value and became inappropriate for use as a result of technological progress. The rest is in full operation.
• As to IBQP-PR's technical capacity, the client satisfaction survey presented a positive evaluation.
• As to the seminars and trainings implemented by IBQP-PR, the results were satisfactory. However, opportunities for improvement were observed in the monitoring of these activities.
• As to the consultancy services implemented systematically by IBQP-PR, the resulting increase in productivity proved that the results were good. Even so the fact that many clients did not emit an opinion shows that there is still a margin for improvement.
• As to public relations and the promotion of productivity improvement that were systematically implemented by IBQP-PR, the high degree of satisfaction in the companies surveyed shows that the expected results were achieved.

3-5 Recommendations

Based on the results of the field study and seeking to correct what was found lacking, the following recommendations were presented to IBQP-PR:

a) IBQP-PR needs to focus more attention in the dissemination of its work within and without the companies it has been acting in. This is because in spite of the high degree of satisfaction (85.71%) in the "communication and public relations" variable, the fact is that ever since the companies were first contacted only the people directly in charge of monitoring the works of IBQP-PR were aware of its action.

b) Some variables presented results that when analyzed as a group may help to explain why the "positive effects" could only be confirmed by 21.43% of the companies surveyed (in this variable the degree of satisfaction and dissatisfaction were the same). A comparison of the results of the evaluation (satisfied = 21.43%, dissatisfied = 28.57%) and acceptance (satisfied = 35.71%, dissatisfied = 7.14%, no opinion = 57.15%) variables shows the high number of organizations that felt insecure in giving a direct opinion (be it satisfactory or not). This leads us to conclude that there is still a large margin for improvement in communications between IBQP-PR and its customers.

c) IBQP-PR needs to get better organized. This statement results from the fact that although the Institute was requested to deliver a list containing the corporate name, trade name, complete address, up-to-date telephone number and e-mail address of its client companies to the consultancy hired by JICA with considerable notice (02.02.2004), this list was only handed in on 02.10.04, one day after the arrival of the consultant in the city of Curitiba (PR). The list was furthermore incomplete. It included only trade names without mention of addresses or e-mail addresses and, in 10.71% of the cases, without telephone numbers. This percentage may have compromised the statistical significance of the sample. Why an organization as important as IBQP-PR doesn't keep a detailed list of its clients remains a mystery.

d) IBQP-PR needs to develop systematic criteria to evaluate its work with maximum urgency in order to, as Socrates put it, "know thyself". This will allow IBQP-PR to see its pros and cons with greater clarity and correct what is wrong before each new contract. In other words, this is the important path for IBQP-PR to reach the desired level of EXCELLENCE which according to the results of the study there is no doubt that it deserves.

e) Performance evaluations need to take place at regular intervals. Comparing the evolution or involution of specific work elements becomes impossible when time intervals vary between two months and two years. Even things that stay the same can represent different things for the company at different times. A performance evaluation can therefore be defined as a process that seeks to evaluate the results of the work accomplished by the company's employees and the results of the company itself through continuous and systematic monitoring, taking in consideration individual and organized characteristics in order to promote institutional development.

f) It is important to note that even though the "costs" factor was judged to be satisfactory by 64.29% of the companies surveyed, this variable should undergo periodic revision in order to remain as low as possible for the clients. After all, in a nation where economic crises often force organizations to choose their partners according to price rather than quality, reducing the costs of services may contribute towards a significant increase in competitiveness.

3-6 Lessons Learned

Even though IBQP-PR's work was approved by most of the surveyed companies due to its IMPACT or SUSTAINABILITY, it is important to keep in mind that in order to reach EXCELLENCE it is necessary to aim at "zero dissatisfaction". And for this, self-evaluation or evaluation of the company's own performance is an essential factor.
The few companies that expressed some degree of dissatisfaction with the work developed by IBQP-PR (as a whole or partially) probably did so as a result of the lack of a rigorous self-evaluation system within IBQP-PR. After all, as mentioned previously, what seems perfect in one organization may not have the same effect in another one. In addition, what brings successful results to one company may bring unfavorable results to another company at another time.
Everything is a matter of context. A performance evaluation undertaken by the organizations in which services were rendered is a precious instrument in order for IBQP-PR to adapt to each new context and scenery
This study represents a growth opportunity as much for the evaluator as for the evaluated. By gathering and analyzing data the reality of the institution surfaced. The actions that resulted from the study sought to optimize individual and group work, redirect any deviations, develop potentials and take advantage of the opportunities to improve professional performance, thus contributing towards the growth of individual professionals and of the institution as a whole.

3-7 Follow-up situations

These don't apply to this consultancy