STAGE 6. Defining the evaluation plan

What steps should be completed?

Step 1: Select the type of evaluation

PROCESS EVALUATION INDICATORS

There are different indicators to measure how an intervention has been developed, what has happened in the implementation, how much and what population has participated and how the intervention has been valued. The indicators can be defined by taking the following questions as reference:

What? In the process of an intervention we usually measure its level of realization, acceptance, coverage and quality. Therefore, the evaluation indicators must be consistent with the action plan. A frequent mistake is to confuse the process indicators with those of outcomes. For example, the number of people attending an activity is a process indicator since it measures the scope (coverage) of it. However, many reports of interventions (erroneously) document the number of attendees to the activities or the number of people assisted as an outcome indicator. This is because, sometimes, it is thought that the ultimate goal of an activity is to reach the population, when in fact, this is a necessary condition but not enough to make it reach the expected changes.

To be able to generate and/or select the most appropriate process indicators to evaluate an intervention it is useful to know: a) the information that can be provided by each indicator, b) the procedure for collecting data on it, and c) the keys to interpret it. With this information, you can make a balance between the most suitable and the most affordable indicators. One Step @ a Time provides you an informative table on some frequently used indicators. It may also be based on that information to determine other indicators not included in the table. +

How? To collect information about the process, quantitative methods as well as qualitative methods can be used. Among the most common quantitative instruments are the questionnaires (self-registers) aimed at the people who implement the intervention (to assess the level of achievement) and questionnaires (self-reports) directed to the target population (e.g.: to know their degree of satisfaction). On the other hand, other strategies such as interviews or non-participant observation (qualitative methods) also provide valuable information on compliance and satisfaction with the action plan of the intervention.  

When? The process is evaluated as the intervention is implemented (at the end of each action or the entire plan of action). The information to know how the intervention develops is compiled as the actions of the project are implemented. That's why data collection tools have to be available in the implementation phase. In addition, some of the information that is collected can be already analysed during the intervention, thus making it possible to make adjustments to better adapt the intervention to the needs of the population.

Where? The data to analyse the process of evaluation are normally collected in the scenarios where the intervention takes place and the main informants are the population (target or intermediary), in addition to the applicators themselves.

How much? In the case of the process evaluation it is advisable to establish criteria of intensity or doses of the intervention, as well as minimum levels of satisfaction on the part of the population. It is advisable to establish the parameters of intensity and quality that are considered acceptable for the intervention to provide changes in the population. For example, it may occur that in an intervention the actions envisaged cannot be developed. If this happens, it is advisable to have references that indicate the minimum and indispensable level to be covered to consider that the intervention has certain quality assurances. It is easier to locate these references if a previously evaluated intervention is applied. In the event that no scientific evidence is found in this regard, expert consensus can be invoked.

Who? Usually, the people who are responsible for compiling the information about the process are the same ones that implement the actions, or, external evaluators, that register and provide more independent information. The profile of the people who are in charge of analysing and interpreting the information depends on what type it is (qualitative or quantitative) and the level of complexity of the intervention.