Keeping it real: co-creating the evaluation of co-production

This blog is written by Lizzie Cain, reflecting on the co-creation of a draft evaluation framework for the UCL Centre for Co-production in Health Research.

What do you think of when you hear the word ‘evaluation’? Having to fill in long questionnaires? Feeling scrutinised and judged? Or a useful, inclusive process of reflection, learning and celebration? It’s my job to try and make sure it is the latter. I’ve been involved with the Centre for Co-production’s evaluation since the beginning, but I’m delighted to have been appointed to join the team in a more official capacity from July. In preparation for this exciting challenge, I’ve been exploring where the evaluation has taken us so far, and where we’re heading next.

Getting started

Evaluation enables us to question our assumptions about what is working and what needs to improve, so we can build on successes and make a real difference. For evaluation to be meaningful, it should also be led by the concerns and priorities of everyone involved. So when we started thinking about evaluating the first stages of the Centre for Co-production in Health Research this time last year, one thing was certain – it was going to be co-created!

We also knew that an evaluation strategy and framework of some kind would be needed to provide overarching guidance and direction, thinking about both the process and impacts of co-production especially in relation to our Phase 1 Pilot projects. But as so much of the Centre’s activity is being co-produced as we go, what does evaluating this look like in practice?

Working with Just Ideas

It looks like getting help! The Centre put out a tender for evaluation support and collectively decided to appoint Just Ideas to lead the co-creation of the evaluation strategy and framework and at the same time opened up the process of co-creation to anyone who wanted to be involved.  In total, 50 people (who occupied one or more of the roles of researcher, student, community member, practitioner) ended up getting together regularly over 6 months to work through ideas, test out concepts and identify opportunities.

This was such an important aspect of the process. Without a truly diverse range of voices in the room, the evaluation framework would have looked very different. For example, Just Ideas quickly stopped using the term ‘evaluation framework’ and instead asked ‘What would success look like?’ And that was success not just through the lens of funders or institutions, but for the Centre collaborators, partners and those who are interested in its work. Evaluation should be meaningful and transparent to those who are involved, not simply a tick-box exercise which is kept behind closed doors.

Just Ideas also worked very closely with the Pilot projects to evaluate their activity in a way that worked for them, as well as contributing to the bigger picture. They developed a 3 step self-evaluation method: a planning session where the Pilot teams identified what they wanted to achieve with their funding, created a simple narrative about their starting point, and identified some progress indicators; selecting one aspect of their activity for an in-depth ‘spotlight’ evaluation; and finally, taking time to reflect on the process of co-production itself.

What did we learn?

“It took more time to ensure that all those involved were clear about the shared aims; we needed to surface differences in understanding and opinion and come to mutually agreeable decisions.”
Pilot Project team member

With the help of Just Ideas and the experiences of our Phase 1 Pilot projects, we have a lot of learning about how to co-produce – what really helps, what can be difficult, how to facilitate decision-making, and much more.

“The lack of anyone specific person being ‘in charge’ which, while in some ways a positive thing to promote equality, made it hard to know ‘who was doing what and when’ when trying to balance providing a co-produced study within the cultural context of the traditional top-down clinical research approach.”
Pilot Project team member

Although the Centre is still at a very early stage of development, some positive outcomes are already emerging. They range from specific project achievements – for example, developing recommendations to improve communication between adults with mild to moderate hearing loss and health professionals – to wider developments, such as the creation of new partnerships and stronger relationships, as well as improved community awareness.

“It is striking that many participants independently commented on how much they value what they have learned during the pilot. The learning has helped to empower their own health management and interest in participating in projects.”
Pilot Project team member

Finally, it wouldn’t be a Centre for Co-Production evaluation if it didn’t involve some reflection on the reflections! Just Ideas left us with their thoughts about how their role developed during the evaluation process, and some  questions about the nature of co-creating an evaluation of co-production. We’ll be taking those with us as we move into the next stage in the Centre’s development.

So where are we now?

Good question! We’re at a point of transition as a Centre, with funding open for our Phase 2 Pilot projects (apply here!), plans to co-create a website and brand, an online resource in the making, and, of course, a draft evaluation framework ready to be tried out.

The framework is made up of 5 ‘success factors’ underpinned by progress indicators. Those factors were identified by our 50 evaluators as the most important enablers for the development of the Centre and its activities in a genuinely co-produced way. They are (in no particular order):

1. How we communicate
2. How we facilitate funding
3. How we work together
4. How we organise ourselves and make decisions
5. How we learn

We’re looking forward to working with whoever wants to be involved to put all this learning into practice,  take the evaluation forward and develop our understanding further. There’s lots to co-create, from translating the success factors into a more practical plan, to deciding what evaluation methods we could use (e.g. video stories, visual tools), and how to communicate this activity more widely. We’re also going to be thinking about evaluating for the ‘bigger picture’ of the Centre’s future.

Not much to be getting on with then….

Get involved!

If you’d like to find out more about the evaluation process so far then you can email Niccola, Project Manager for the UCL Centre for Co-Production, to request Just Ideas’ full report: n.pascal@ucl.ac.uk. You can also contact Niccola to be added to the Centre’s mailing list for future co-creation opportunities.

And finally, make sure you keep an eye on this blog to stay up to date with everything that’s going on. We look forward to sharing it with you!