Ofsted's use of Artificial Intelligence: how smart is it to automate risk assessments?

Melanie Ehren.
Ofsted has come under attack for its collaboration with the Behavioural Insights Team for using machine learning to identify failing schools. According to several sources (BBC and Matthew Reynolds), BIT has been trialling machine learning models that can crunch through publicly available data to help automate Ofsted’s decisions on whether a school is potentially performing inadequately. The algorithms use information on number of children on free school meals, how much teachers are paid, the number of teachers for each subject, and particular words and sentiments in reviews of schools submitted by parents on the Ofsted-run website Parent View.
As Ofsted’s head of risk assessment (Paul Moore) explains:
‘For a number of years Ofsted have risk assessed maintained schools and academies. This risk assessment is used to help put inspection resource into schools where it is most needed. It’s important to note that it has never been used to pre-judge inspection grades. The risk assessment model has evolved over the years, as inspection frameworks and accountability measures have changed.
The latest risk assessment development work is a continuation of our aim to continually improve our models. It influences how we plan our inspections, but inspectors are not given the findings of the risk assessment to avoid it having an influence on the inspection itself, which is based on the evidence they gather on-site. Over recent months Ofsted have been investigating whether a machine learning approach to predicting school decline could be a useful risk assessment development. The Behavioural Insight Team have been assisting us in upskilling our staff, and providing expert advice on machine learning techniques’.
Paul is keen to emphasise that the role of risk assessment stops at the point that Ofsted selects which inspections to carry out. Once an inspector arrives at a school, for example, all the focus is on the evidence that they are able to gather in person and in dialogue with the school’s leaders—the risk assessment plays no further part.
Automating risk assessments seems an intelligent approach to using scarce inspection resources more efficiently. But the recent critique begs the question of whether this combined use of data and human judgement is an actual example of ‘intelligent accountability’? Crooks (2006) provides an extensive description of ‘intelligent accountability’ in saying that it is a system which:

• preserves and enhances trust among key participants
• involves participants in the process
• offers participants a strong sense of professional responsibility and initiative
• encourages deep, worthwhile responses
• provides well-founded and effective feedback to support good decision-making and
• leaves the majority of educators more enthusiastic and motivated in their work.

The focus on data and machine learning particularly supports the ‘decision-making’ part of the description when risk assessments are used to inform inspection scheduling. Data and risk assessments however don’t tell us much about how accountability can motivate people to learn and improve. Mechanistic assessments are generally far removed from the people who have to learn from these assessments and are therefore not the most motivating or best ways to support learning and improvement. This is a point well understood by Ofsted in saying that inspectors are the ones who make the ultimate decision on a school’s quality, based on on-site evidence collection.
Ofsted’s emphasis on the human factor in making decisions on school quality rightfully takes into account that schools are not factories where children with high learning outcomes are produced according to prescribed standards scripts. At best, learning requires co-production between children, teachers, and their parents, where they are engaged and committed to learning and teaching and have an opportunity to explore and nurture individual talents and interests. Risk assessments however ensure that a basic standard can be safeguarded with limited resources. But it’s ultimately the interaction between schools and inspectors about actual achievements which will generate new ideas for improvement that can lead to real learning.