Dismantling Mantras

I’m not a GP or an academic because I like to conform, so it should come as no surprise that I can’t hear a mantra without wanting to challenge it. The quality improvement world is full of popular wisdoms, rehearsed and re-rehearsed by its enthusiastic followers. How about this one: “Data should be used for improvement, not for judgement.”

No shortage of experts in the field have differentiated between the characteristics of data used for improvement, accountability and research purposes. They tell us that data used for improvement can be ‘good enough’, that it is used to indicate rather than to reach definitive conclusions, that bias can be tolerated and that control charts allow us to attribute outcomes to interventions. In contrast, we are told that data used for accountability purposes need to be more rigorous, that it is used to reward or punish, that bias needs to be minimised. And for completeness we are told that data used for research purposes needs to be of top quality, as devoid of bias as possible and that analytical or inferential statistical tests are used to attribute outcomes to interventions. It all sounds very neat and reasonable.

But like all mantras it has a political purpose. Differentiating between improvement and judgement allows improvers to position themselves on the side of the angels. Improvement is benign, positive, enabling; accountability is malign, negative and damaging. Improvers make reasonable judgements, failure is not an option; holding people to account is unreasonable, done by people who don’t even understand the basics of common and special cause variation, never mind the intricacies of statistical probability.

Where the improvement world has come from is understandable but I don’t think that their position is sustainable. Nearly a quarter of a century after quality improvement techniques were introduced into the health sector from manufacturing industries it should be main stream but it isn’t. In most organisations only a small proportion of enthusiasts are engaged with using systematic and data-driven improvement activities. There are many reasons for this and scepticism about some claims of success that are made using poor quality data is one of them. Allowing questionable data to be used in questionable ways does not help to place an improvement philosophy and methods where they need to be – centre stage.

Anyway, it’s naive to suggest that we shouldn’t judge. I’m writing this blog on a train and the man sitting next to me is wearing a very dodgy yellow shirt. That’s a judgement. I’m sure that others in the carriage think he’s a fashion icon. And now you are making a judgement about my judgement. Judging is part of the human condition and what matters is not the judgement itself but the implications of the judgement – and the implications of using data inappropriately are significant.

Rather than service-based improvers, system managers and academics using different data, we should aim for convergence, using data as a common language in a way that allows everyone to focus on a common interest – improving value for individuals and communities.

So, we need to improve the quality of data we use for improvement so that better judgements can be made, and improve the quality of data we use for judgement, so that better improvements can be made. And no, that’s not a mantra, it’s a suggestion.