Evaluation Exchange: Measuring wellbeing with sensitivity

The Evaluation Exchange is a venture between UCL Public Engagement Unit and Aston Mansfield, which sees groups of researchers from UCL working together with a voluntary sector organisation (VSO) in east London to solve an evaluation challenge. Read previous posts in this series.

This week we’re hearing from team Magpie Project, an organisation providing support and advice to mothers and children under five who are in temporary or insecure accommodation in Newham. The research team is made up Paula Oliveira, Emma Laycock, Masuma Mishu and Fran Harkness. In this blog post, Fran tells us about their journey so far.

---

One of the hardest things we’ve had to deal with in our volunteer team of evaluators- apart from learning what evaluation is- is figuring out how to gain evaluation data from the families who use our charity partner, the Magpie Project. We’ve put it off till one of the very last tasks.

Doing a PhD IS stressful. Your findings don’t come out right, supervisor meetings are anxiety-provoking, the printer doesn’t work at the vital moment. But the difficulties that the women and young children attending the Magpie Project are dealing with are a world away. These families are temporarily homeless and attending the center for advice, friendship, and for the simplest reason: a clean warm place for their children to play. I think for all of us, the moments we’ve been sitting on the floor of their one room listening to a Mum speak about how attending the project changed her life put everything into perspective.

But those moments didn’t come until three months in. At first we were a bit unsure of speaking to the Mums for the evaluation. The Magpie team had told us that they wanted us to design a tool to evaluate whether they’re improving the families’ wellbeing. However, all the charity literature online says how unethical and complicated it is to attempt to collect evaluation data from people who are vulnerable or have been through trauma. In PhD student terms we think, ‘Ok we just need to extract some data’. In human language that would be asking people who have gone through great difficulty, “Just how awful do you feel on a scale of 1-5”?”. Most of us hadn’t worked with ‘real people’ (read: non academics) for a long time or ever so the idea of these distressing questions had us longingly running back to our pipettes and computers.

So how have we done it? Well. We first worked for a long time on a theory of change model for the project. This was to zoom in on the true aim of the project so we know exactly what needs to be evaluated and why before we started trampling all over people’s feelings. There’s no use data collecting something we don’t need.

Our theory of change model showed us that it wasn’t imperative to measure wellbeing directly as we could map out that wellbeing should have been achieved if all the other components the charity provide in the model are fulfilled. We read other charity evaluation reports too and found that many had followed a similar path to avoid causing any more distress to their vulnerable clients.

We realised the importance of talking through challenges with our charity team: whilst we were scratching our heads about how to measure wellbeing, they had also come to the realisation that they didn’t need as much data on it as they had previously thought. Less probing needed! Phew. We had been on the same page without realising it.

And most importantly of all, we asked the Mums what they thought. We sat on the floor next to them playing with their little ones and asked exactly how they thought we should evaluate their wellbeing. To them it seemed perfectly logical that we would want to evaluate whether Magpie was helping them to feel better. Hearing their opinions rather than making assumptions about them helped us to revise a very simple but effective picture tool which Magpie are going to start testing soon.