Gathering robust evidence
What is meant by robust evidence of change & impact?
Having recently measured our own education pet care talks we were able to measure knowledge, attitudes and empathy and can show a significant increase in all areas. Knowledge was the highest and we are working on improving the other two.
Capturing the wider impact of an intervention requires robust evidence of change, and this was considered to be one of the most significant challenges facing animal welfare education specialists. This type of evaluation focuses on objectively assessing the outcomes and mechanisms of change that have been identified in the logic model.
This is the most intensive type of evaluation, requiring the most planning, resources and skills, which perhaps explains why many organisations do not carry out evaluation of this kind on a regular basis. Some practitioners working with children and young people are not familiar with the assessment tools and the process of collecting and analysing data of this kind. However, it is the most powerful demonstration that an intervention is making a difference; enhancing the knowledge, attitudes, skills and/or behaviours of those participating. Some funders also require this type of evidence in order to invest in an intervention or service. It should be noted that some outcomes are easier or quicker to achieve than others.
Types of information that might be included:
- Standardised measures – usually questionnaires that have been designed to measure the outcomes or change mechanisms you are interested in, but may also include other tools like observations or interviews.
- Routinely collected data – often relevant information is already being collected by other people (in the local community or nationally). Accessing this information can provide valuable evidence of impact. This may include records of animal cruelty cases or offences, the number and type of phone calls to rescue charities from children, the extent of positive behaviour examples, such as charitable work by children or schools.
Many organisations are keen to use standardised measures, but are unable to access them or do not know how they should be used/analysed. Accordingly, we have gathered together some examples of published measures that can be used freely with children and/or young people to capture some of the outcomes that animal welfare education professionals want to see.
Depending on the goals of the intervention, a pre-test, post-test, and delayed post-test can be used to demonstrate immediate and longer-term impact on participants. However, it is important to note that collecting evidence before and after an intervention cannot tell you with confidence that it is the intervention that has caused a change. Something else may be responsible. To establish this, as we indicated earlier, it is very helpful to compare the children who are participating with another group who are not (a control group). Ideally, this would be a group who are on a waiting list to start the intervention, but if not, the control and intervention groups need to include participants with similar characteristics in order to make them comparable.
An example is currently being developed.
Other types of outcome/impact evaluation
What else do I need to consider?
We have discussed the main steps in evaluating an intervention (What do do) and will shortly provide a worked example. However, we have also outlined how to go about this process to ensure the intervention and evaluation are the best they can possibly be (How to do it). We recommend careful consideration of four facets, and hope they will be easy to remember with the acronymn CoRES:
Collaboration/identifying expertise & deciding on shared values, outcomes & approaches
Researching, adapting & developing appropriate resources, delivery methods & assessment tools
Establishing what is/is not, working well, making adaptations/refinements and re-assessing
Sharing findings & issues, including the challenges & what has not worked
How should I develop the intervention and evaluation? |
|