What is a Process Evaluation?

For the Platform a Process Evaluation (PE) is a systematic analysis of a program carried out to identify how the processes and activities lead to the achievement of the results defined in the design, and to detect bottlenecks and best practices that have affected the implementation of processes and activities, so as to offer recommendations and areas of opportunity for improvement (Mendoza, Moreno-Sanchez and Maldonado, 2015). Its implementation is carried out in parallel with the program implementation, focusing on those processes that lead directly to the fulfillment of the objectives or purpose of the program.

Given the nature of the analyzed object, this is an assessment that uses qualitative information primarily from, for example, structured interviews to entities’ staff (program responsible and implementers) and participants. It can also employ quantitative data, resulting from structured surveys, for example, seeking to estimate numerical indicators. These indicators can be about the processes’ effectiveness (to determine whether the process execution contributes to the achievement of the objectives) or their quality (to identify perceptions of the performance of the implementing agencies and satisfaction of program beneficiaries)…

A Process Evaluation is very useful for designers and implementers and even for other evaluators interested in analyzing some other facet of the program.

Process Evaluation allows us to have an understanding of how the design program is actually running in the field and whether the design facilitates or hinders the achievement of the proposed objectives. The analysis contemplated in the evaluation, identifies bottlenecks (practices or procedures that creates obstacles for the achievement of goals), best practices (novel, replicable and sustainable in time initiatives that strengthen the program), or external factors (non-related aspects that affect the program execution). Similarly, by performing a critical review of the program’s processes, the analysis in the evaluation identifies areas or aspects for improving implementation in the short and medium term.

Moreover, other assessments that can be made of the program (e.g. product, outcome and cost-benefit evaluation) depend partly on assuming that it is well designed and implemented. Thus, a PE allows us to contemplate how restrictive the assumption made is, and to give a further explanation of these other evaluations’ results.

How is a Process Evaluation done?

1. The hypothesis construction

Orients the evaluation exercise. The hypothesis should be clear and should guide goal setting.

2. Goal setting

Considers objectives in line with the hypothesis that respond to the questions What? How? and Why? The number of specific objectives should be no more than five or six.

3. Defining the scope

Begins by identifying programs processes and sub-processes, and by selecting those that are substantive (those that directly lead to the fulfillment of its objectives) in order to define those on which the evaluation will focus.

4. Development

It is the core of the evaluation and is carried out in parallel to the implementation of the Graduation Programs.

It requires the identification of the key moments in Graduation Program implementation, which can be:

  • The program start, when it is necessary to collect baseline or context information;
  • The program end, when information for determining whether products and expected results correspond to those obtained is collected
  • Intermediate moments when core products to achieve the program objectives are distributed

The results are the input for the formulation of recommendations for future programs’ scale ups and implementations.

 

4.1. Detailed description of the program

This is carried out in order to understand the operation and functioning of the Graduation Program in detail. It includes four aspects:

  1. Factual and contextual analysis.
  2. Value chain construction.
  3. Construction of the actors’ map and identification of responsibilities in the value chain.
  4. Detailed schedule of the program.
4.2. Effectiveness analysis

This seeks to determine whether the objectives were achieved. It includes three activities:

  1. Define the indicators.
  2. Estimate indicators.
  3. Analyze the indicators.

It is necessary to formulate indicators for each substantive sub-process. These indicators are generally quantitative, however, the values must be framed and they need to take into account the results obtained from qualitative data collection.

4.3. . Quality analysis

This allows the identification of the performance of the implementing agencies and participants’ point of view.

The Platform’s quality indicators are defined to measure the usefulness, timeliness (right time), sufficiency (required amount), and the satisfaction of the provided inputs and the developed activities in the different processes, from the supply (implementing entities) and the demand (participants) perspectives. Indicators can be quantitative or qualitative.

This includes three activities:

  1. Define the indicators.
  2. Estimate indicators.
  3. Analyze the indicators.

The resulting information must be contextualized and explained from the data triangulation obtained in other steps.

4.4. External factors analysis

This analysis seeks to identify, in general and contextual terms, different external factors (some of them not controllable), which may affect the development of processes and results of the Graduation Program.

This analysis aims to strengthen the formulation of recommendations for improving the Graduation Program.

It is outlined using the value chain in each of the sub-processes.

4.5. Bottlenecks and best practices identification

This seeks to identify, describe and understand the best practices and bottlenecks that arose during the implementation of the Graduation Program, analyzing the causes and consequences of each of them.

This analysis aims to strengthen the formulation of recommendations for improving the Graduation Program.

It is outlined using the value chain in each of the threads.

5. Formulation of conclusions and recommendations

This seeks to provide a comprehensive concept of the program and proposes strategic planning guides that allows corrective actions in the short and medium term.