Evaluation Exchange: Anchoring practice in user experience

The Evaluation Exchange is a venture between UCL Public Engagement Unit and Aston Mansfield, which sees groups of researchers from UCL working together with a voluntary sector organisation (VSO) in east London to solve an evaluation challenge. Read previous post in this series.

This week we’re hearing from team Caritas Anchor House, in this blog post written by Joana Neves Dos Reis from UCL and Victoria Palazzo from CAH.

---

What do UCL researchers and charity workers have in common? We both want our work to have a positive impact on society by using solid evidence about ‘what works’.

The Evaluation Exchange, organised by UCL and Aston Mansfield, brought UCL students and staff from the homelessness charity Caritas Anchor House (CAH) together to first co-develop a tool to evaluate CAH’s services and then incorporate CAH residents’ voices into decision making at the charity.

The CAH team (minus one member- Kathy from CAH).  From left to right; Joana, Femi, Anne, Victoria, Aradhna and Jean. 

CAH is based in the London Borough of Newham, where one in 25 people are homeless – the highest in Britain.  By providing accommodation, education and guidance for single homeless adults, the charity’s work aims to ensure that all those who walk through their doors grow in confidence, and move towards leading independent, self-fulfilling lives. With its limited resources it is doing its best to become more effective.

Located in Canning Town, Caritas Anchor House supports up to 250 people who are experiencing homelessness each year

 

Our challenge is to capture CAH residents’ honest and constructive feedback on CAH’s services. This is easier said than done. Feedback is collected throughout the resident journey, however it only tells part of the story.  There are many dimensions and contexts that have to be accounted for when evaluating. Researchers knew the ‘how to’. Charity workers knew the ‘what’.

Our first meeting focussed on articulating that and finding common ground. Researchers had to understand the system to evaluate. Charity staff needed to identify what was going to be evaluated. Our first task was then an exercise of empathy, of imagination: determine what to ask residents. Imagining what is important to someone in completely different circumstances from ours is a difficult task. We realised we needed residents in our team, former and current, so we welcomed Valerie and Biruk. They were able to keep us on track to what is important for residents. They are our experts by experience.

Staff at Caritas Anchor House support residents by providing safe accommodation and access to new opportunities

 

The second task was to formulate questions to capture everything the residents value about CAH and what they would like to see improved. The questions needed to be open enough to capture the whole story but to also give us the clear answers that could be translated into actions. We decided to do focus groups with carefully selected representatives of current and former residents.

This method presented us with a number of challenges including interviewing time, anonymization and data analysis. The researchers were not satisfied that this was possible and recommended that we change our method to use Nominal Group Technique (NGT). NGT is a method of consensus decision making where individuals answer questions in a group setting and then ‘vote’ for the answer that they most agree with. This would save us time whilst still producing quality results. This method required us to reflect on what we really wanted to evaluate and how to ask a reduced set of questions to get the most effective answers.

We succeeded in summarizing our questions and plan to start interviewing residents in January. The feedback we gather will be used to then develop a tool that integrates user voice into decision making at CAH. This is an important step in empowering residents, and we hope that by learning to listen more effectively we can empower residents and improve their experiences and influence in the organisation. We hope that it will help them see that change is possible.