STAGE 6. Defining the evaluation plan

What steps should be completed?

STEP 2: Define the indicators and evaluation methodology

An indicator¹ is a unit of measurement that reports on the transformation of a phenomenon. It is therefore used to verify the changes that have taken place in this phenomenon over a period of time or after an intervention. In the context that concerns us, indicators help us to measure and reflect in a simple way the changes made in the intervention objectives, and therefore the target population (on their behaviour, knowledge, attitudes, environment, etc.) in a particular period of time (usually between the beginning and the end of the intervention)2, and to assess the action plan.

Outcome indicators are related to the effects of the intervention on the specific objectives of the target population, while the processes indicators are related to the products of the intervention (structures created, materials published, etc.) in relation to the intervention's actions.² The table below illustrates the logical relationships between these elements.

Based on: EMCDDA. (2010). Prevention and Evaluation Resources Kit (PERK).

 

A common error, which should be avoided, is to define or select the evaluation indicators too late, when the intervention has already started or even once it has finished. It is imperative to define them before starting the intervention.

To define the indicators, it may be useful to ask a number of questions, the responses to which will shape the methodology of intervention evaluation:

What? | Determine what to measure and select the data to be collected. Implementation and efficacy are two of the key pieces of information about an intervention that are useful to know. Information on implementation is obtained from process indicators +, while efficacy is measured using outcome indicators. +

Various indicators are usually used to evaluate the process and the results of an intervention, as there are usually no indicators that can fully reflect the quality of the implementation or the efficacy of an intervention when used alone.

There are usually several groups of stakeholder with an interest in a drug demand reduction intervention (planners, administrators, financers, target population, etc.) so, when selecting what to evaluate, it is useful to consider and incorporate the visions and expectations of the various parties involved. However, the more indicators are used, the more complex is the evaluation, but it will also have more information. A possible solution would be to select the most important issues and prioritise them according to their relevance. It is also very important that the indicators (especially results indicators) are as specific as possible, as well as measurable, suitable, realistic and of a suitable duration (they must allow changes to be observed in the time periods defined by the evaluation).²

How? | Specify the methodology for collecting information on the indicator. The methodology includes methods and tools for data collection, analysis and interpretation. The indicators that have been selected will illustrate whether it is more appropriate to use quantitative methods or qualitative methods of analysis. It is often advisable to use mixed methods that combine the tools and techniques of both, as they provide complementary information and give a fuller view of the intervention.

When? | Determine the time to collect the information. The data that is generated can be collected while the intervention is being implemented and/or prior to the intervention and/or after the procedure has been completed, depending on the evaluation design used.

Where? | Determine sources, informants and situations where information is to be collected. Sources and informants must be accessible and of a proven quality to collect information. It is better for an intervention to include few indicators, but informative and accessible, rather than many but hard to access. If it is costly to obtain the information, it will be more likely to be abandoned and the indicator not to be measured.

How much? | Report the expected level of change. An indicator should be measurable and interpretable. Therefore, references should be included in the action/evaluation plan to help verify whether the intervention is being carried out according to the initial forecasts and whether the results that are achieved are as expected.

Who? | Determine the persons responsible for designing the instruments used to collect the information, analyse it and interpret the results. The number and characteristics of the professionals responsible for collecting and analysing information relating to the indicators depend on the complexity of the intervention and the type of evaluation.

 

References:

1 Adapted from Alonso C, Salvador T, Suelves JM, et al. (2004). Prevención de la A a la Z. Glosario sobre prevención del abuso de drogas [Prevention from A to Z. A drug abuse prevention glossary]. Madrid: Centro de Estudios sobre Promoción de la Salud.

2 Adapted from European Monitoring Centre For Drugs And Drug Addiction. (2010). Prevention and evaluation resources kit (PERK). Luxembourg: publications office of the European Union