The first meeting of Evaluation Exchange, the peer support group for self-evaluators facilitated by IRISS (Institute for Research and Innovation in Social Services) and ESS (Evaluation Support Scotland), was held on Wednesday 31st October and was, by all accounts, a really successful, productive meeting.
Our ten peer-evaluators come from the third and statutory sectors and are working on evaluating projects in fields from substance misuse to supporting people with communications difficulties. You can find out more about them here.
The day was split into three main sections: establishing the group and its purpose, general discussion and training around evaluation and specific work on individual evaluation projects.
Getting to know you…
From the outset it was clear that everyone was really engaged with and enthusiastic about the peer support aspect of the group, with people talking about being excited about being open and honest, working in partnership
, discovering answers together and hearing from new people and new perspectives. Challenges were also acknowledged mainly under the banner of keeping focused and finding the time to prioritise evaluation amid competing demands. Nevertheless, we felt that it was important to make time outside of our meetings to reflect on what we are learning, to check in with each other and to work on our evaluations.
We also agreed some criteria about the success of the programme. We agreed that our participation would have been successful if:
• We have a robust evaluation
• We have an evaluation system in place
• We can ‘crack outcomes’
• We have new ideas to take forward in our organisations
• We can streamline monitoring and evaluation
• We are energised and excited about evaluation
• We can motivate or enthuse our colleagues
Evaluation – so what?
Members of the group had a number of reasons for wanting to undertake evaluations including:
• Showing how well we are doing
“We need to show how good we are”
• Show we are making a difference to people’s lives
“I feel very strongly, we KNOW it’s making a difference so we want it to continue. So we NEED an evidence base.”
• Seeing where we might need to get better
“[Evaluation] stops you assuming what is working well for you. You can get complacent when everyone says it’s a great project but this helps you break it down and shine a light on what’s really going on.”
However, for everyone in the group (perhaps unsurprisingly since they had applied to take part in a peer support network!) evaluation was seen to be really important.
“There’s no getting away from it, especially in social work, we work purely on an evidence base”
“I’m passionate about evaluating our work.”
During the session facilitated by Tom from ESS, the So What? game particularly struck a chord with the group. The game is designed to help you decide whether or not an issue you have decided to evaluate i
s really an outcome – unless you can answer the question “So what?” convincingly then you are not measuring an outcome – but rather an output or an indicator.
This process generated a lot of discussion that touched on some of the more challenging aspects of evaluation. The group talked the difficulties in making connections between different levels of outcomes – personal, organisational and those from funders or government – and in turn this led on to the fundamental question of who the evaluation is being done for – is it for our own organisations to help us improve the services we provide or is it to meet reporting requirements or secure funding? And what is the correct balance between these needs? This also sparked discussion around the challenges of measuring ‘softer’ outcomes and how best to really include the voices of those we are working with in evaluation.
“Makes you wonder if it’s all about the people you work with?”
“Evaluations should make sense to the people we work with – that should be at the centre. But is it? Or is government/funders? Reconciling service and service user outcomes is difficult, it’s a process.”
Specific projects
Each member of the group had come prepared to present for 5 minutes on their project, which we did before splitting into three smaller groups with the purpose of clarifying evaluation questions, deciding what is needed to answer those questions and how Evaluation Exchange can help. The success of this part of the session was mixed. While the work on individual projects led to interesting discussion and reflection, we felt that at this stage clarity on evaluation questions was not wholly achieved. Given the different stages of thinking that the group members began at, this was likely inevitable. The facilitators were also wary of their role in the small groups being perceived as ‘the expert’ and this undermining the crucial peer support aspect of the project.
What are we going to do next?
We worked with the group to identify priorities for future sessions. We agreed that at each future session we would split the time between working on individual projects and addressing key evaluation challenges. Throughout the first session a number of issues and challenges in relation to evaluation were raised and, for the next session on 5th December, the group has agreed to concentrate on:
• Evaluation methods – traditional and creative
• Involving service users in evaluation
And in between times we will, of course, be thinking about what we learned at the first meeting, keeping in touch via Basecamp – our shared online space – or however else we choose, and working on our evaluation projects. We will also be reflecting on the following two questions:
1. How good is my organisation at evaluation?
To help you answer that question you could consider these 4 statements
• We know what to evaluate because we have clear outcomes for our organisation or project
• We have appropriate systems for collecting information / data about our outcomes
• We can analyse and report on our outcomes
• We use learning from evaluation in our ongoing work
2. If your participation in Evaluation Exchange were to have an impact on your organisation (not just you) what’s one thing that might change or be different in your organisation or colleagues?
We invite you to do likewise!