Rebecca Allen
Today, Jay Allnutt and I published a new piece of analysis (PDF) showing that schools taking on Teach First participants have achieved gains in their GCSE results as a result of the programme. Our analysis tracks the performance of these schools in the first three years after they join the programme and compares them to changes in progress at a set of schools that look identical, except for their Teach First participation in that year.
We make sure this comparison set of schools have the same pupil demographic profile, the same prior levels and trends in GCSE performance, are in the same region of England and are all schools who will choose to join Teach First at some point in the future (formally this is known as a matched difference-in-differences panel estimation). Overall, school-wide gains in GCSE results are in the order of an improvement of about one grade in one of a pupil’s best eight subjects. This estimate is a fraction of the size claimed by the only other quantitative evaluation by Muijs et al (PDF), but is still large enough to be of value to schools.
Like many, until I wrote this research I was sceptical that Teach First participants could possibly have any sort of transformative effect on schools. The academic research tells us that we don’t know much about what good teachers look like before they join the profession. We just know they are likely to be relatively ineffective in their first year of teaching, compared to their second and subsequent years. So, to my mind, setting up a scheme that legitimises exit from the profession after just two years seemed like lunacy.
Furthermore, to someone who trained to become a teacher for a whole year and still found my first year in the classroom extremely exhausting and challenging, it did seem rather reckless to place young graduates into challenging classrooms with so little time to prepare.
But on reflection, I now think Teach First provides important lessons for recruitment to the teaching profession as a whole.
I believe Teach First reminds us of the importance of selecting the right teachers at the outset
Even in the depths of recession in 2010 there were just 3 applicants for each place on a traditional graduate training course (PGCE, PGDE and GTP). By contrast, at the same time Teach First was processing about eight applications for every place. Without needing to make claims about the relative qualities of the respective pools of applicants, it is easy to see how Teach First is able recruit an intake with greater potential, provided their sifting process can do a reasonable job of spotting those with the drive, resilience and stamina needed to succeed in the classroom.
Recent academic papers from the US explain that getting this initial selection of teachers right is critical because, whilst a first year novice teacher is less effective than they will be in year two, the improvement in teaching quality gained through experience is actually relatively modest compared to the very wide variation in teacher quality at the outset. Furthermore, those who are weak teachers in year one improve their practice at a slower rate than others, thus widening gaps in effectiveness in years two, three and four.
Giving graduates an exit path after two years may be an important recruitment device
It is hard to really know whether you’ll love teaching before you try it. The great pleasures of teaching are rather strange compared to other graduate jobs and many 21 year old graduates will have had little contact with young children in recent years. Given this, while many graduates hesitate about training to join the teaching profession (forever?), they may feel more able to join a scheme with a straightforward exit route after two years. If they find they need to (which most don’t), they can walk away with a sense of completion rather than as a ‘failure’ or ‘drop-out’. And the greatest success of Teach First takes place when a graduate who fully intended to move onto a job in the City or in Law completes their two years and finds they are unable to rip themselves away from the delight and gratification of educating the next generation.

Rebecca Allen
David Laws, the Liberal Democrat Minister for Schools, has been making a series of speeches over the past month about “closing the gap” in the attainment between pupils from deprived and more affluent backgrounds. Yesterday, he warned that schools should not be judged as outstanding by Ofsted if they failed to close the gap, a goal that sounds fair and even laudable in principle, but I believe is rather unfair in practice.
The “gap” is the difference in GCSE achievement between the average for pupils who are eligible for free school meals and the average for those who are not. Pupils eligible for free school meals have similar characteristics across schools since they all come from families claiming some sort of benefit. The problem is that the background of pupils who are not eligible for free school meals (FSM) will vary considerably across schools, since the group includes both those with bankers and with cleaners as parents.
A school may substantially narrow the gap by working hard to improve the attainment of their most deprived children, or through the accident of the characteristics of their non- eligible children. I have written before that very deprived schools tend to have very small gaps, a quirk I first spotted as a governor of a school that was struggling to produce strong academic results but was very proud that its FSM gap was zero. All the students at the school came from low income families living on a very large and universally deprived council estate. Some of the families happened to claim benefits that made them eligible for free school meals (not necessarily the poorest), others didn’t or couldn’t. Not surprisingly, the GCSE performance of the FSM and non-FSM pupils in this school were no different, on average, because these pupils were no different in their social or educational background. Nothing the school was doing contributed to this supposed “success” in “closing the gap”.
Social class disadvantage is an important national policy problem but I do not believe that schools should be deciding policies based on the size of their FSM attainment gap. What matters to children from low-income families is that a school enables them to achieve a qualification to get on in life. If a low-income student gets a low quality education from a school, it is little consolation or use for them to learn that the school served the higher income students equally poorly (i.e. the school’s “gap” was small).
As it turns out, great schools tend to be great schools for all children in the school – the statistical correlation between who does well for FSM children and who does well for non-FSM children is very high. Moreover, schools can make a difference to the life chances of FSM children – there are huge differences in attainment for these children across schools, far larger than there are for children from wealthy backgrounds who do pretty well in all schools.
Should the pupil premium be used exclusively for FSM children?
In an earlier speech David Laws threatened to ring fence the pupil premium money to force schools to spend it on their FSM pupils, rather than throwing it into the general pot of schools money. (The pupil premium is the policy tool by which schools are supposed to “close the gap”). I worry about this type of restriction in spending on certain activities, not least because the FSM children are not always from the poorest families in the school – the very act of claiming benefits means some family income leapfrogs those of others that work. More practically, students eligible for free school meals are not segregated into special classrooms, so it is rather difficult to spend the pupil premium on important things such as teaching and learning without the benefits spilling over to other children who sit in the classroom with them.
Free school meals children do not have educational needs that are unique or particularly distinctive. Like other children, excellent teaching and support for their learning is a good place to start. I do not deny that social deprivation in the home spills into schooling and it is clearly possible to target this via attendance incentives, breakfast clubs, homework clubs and tutoring support. But I believe schools would find it morally and emotionally difficult to exclude a child who is not on FSM from accessing these schemes if they could clearly benefit from them.
I hope that my very specific criticisms of an admirable policy do not detract from the urgency of understanding and reducing the size of social class gaps in educational achievement in England. Schools must engage in the evidence on resourcing and pupil achievement and make careful, deliberate budget decisions that improve pupil learning, particularly for those who are most educationally disadvantaged in their school. This may mean spending the pupil premium to retain the best teachers, introduce policies to improve teaching practice or even an “approved” activity targeted very specifically on more deprived children in the school.
However, I suspect that in trying to support the attainment of FSM-eligible children, a school might inadvertently help those not eligible too. Is this such a bad outcome? And if not, why do we use a “gap” metric that “punishes” a school for improving the attainment of those not eligible, as much as it does allowing the attainment of more deprived children to fall behind?

Rebecca Allen
As a quantitative researcher in education I am delighted that Ben Goldacre – whose report  Building Evidence into Education was published today – has lent his very public voice to the call for greater use of randomised controlled trials (RCTs) to inform educational policy-making and teaching practice.
I admit that I am a direct beneficiary of this groundswell of support. I am part of an international team running a large RCT to study motivation and engagement in 16-year-old students, funded by the Education Endowment Foundation. And we are at the design stage for a new RCT testing a programme to improve secondary school departmental practice.
The research design in each of these studies will give us a high degree of confidence in the policy recommendations we are able to make.
Government funding for RCTs is very welcome, but with all this support why is there a need for Goldacre to say anything at all about educational research? One hope is that teachers hear and respond to his call for a culture shift, recognise that “we don’t know what’s best here” and embrace the idea of taking part in this research (and indeed suggest teaching programmes themselves).

It is very time-consuming and expensive to get schools to take part in RCTs, (because most say no). Drop-out of schools during trials can be high, especially where the school has been randomised into an intervention they would rather not have, and it is difficult to get the data we need to measure the impact of the intervention on time..
However, RCTs cannot sit in a research vacuum.
Ben Goldacre does recognise that different methods are useful for answering different questions, so a conversation needs to be started about where the balance of research funding for different types of educational research best lies.
It is important that RCTs sit alongside a large and active body of qualitative and quantitative educational research. One reason is that those designing RCTs have to design a “treatment” – this is the policy or programme that is being tested to see if it works. This design has to come from somewhere, since without infinite time and money we cannot simply draw up a list of all possible interventions and then test them one by one. To produce our best guess of what works we may use surveys, interviews and observational visits that took place as part of a qualitative evaluation of a similar policy in the past. We also used descriptions collected by ethnographers (researchers who are “people watchers”). And of course we draw on existing quantitative data, such as exam results.
All of this qualitative and quantitative research is expensive to carry out, but without it we would have a poorly designed treatment with little likelihood of any impact on teacher practice. We may find that, without the experience of other research, we might carry out the programme we are testing poorly for reasons we failed to anticipate.
The social science model of research is not ‘what works?’ but rather ‘what works for whom and under what conditions?’
Education and medicine do indeed have some similarities, but the social context in which a child learns shapes outcomes far more than it does the response of a body to a new drug. RCTs may tell us something about what works for the schools involved in the experiment, but less about what might work in other social contexts with different types of teachers and children. Researchers call this the problem of external validity. Our current RCT will tell us something about appropriate pupil motivation and engagement interventions for 16-year-old teenagers in relatively deprived schools, but little that is useful for understanding 10-year-old children or indeed 16-year-olds in grammar schools or in Italian schools.
The challenge of external validity cannot be underestimated in educational settings. RCTs cannot give us THE answer; they give us AN answer. And its validity declines as we try to implement the policy in different settings and over different time frames. This actually poses something of a challenge to the current model of recruiting schools to RCTs, where many have used “convenient” samples, such as a group of schools in an academy chain who are committed to carrying out educational research. This may provide valuable information to the chain about best practice for its own schools, but cannot tell us how the same intervention would work across the whole country.
Social contexts change faster than evolution changes our bodies. Whilst I would guess that taking a paracetamol will still relieve a headache in 50 years’ time, I suspect that the best intervention to improve pupil motivation and engagement will look very different to those we are testing in an RCT today. This means that our knowledge base of “what works” in education will always decay and we will have to constantly find new research money to watch how policies evolve as contexts change and to re-test old programmes in new social settings.

Rebecca Allen
Thousands of primary and secondary schools have chosen to convert to academy status (the chart below covers secondary education). A survey by the think-tank Reform showed that financial considerations were the most widely cited reason for conversion, as predicted by many, including The Guardian.
The financial gains arise because academies can directly claim their share of the Local Authority Central Spend Equivalent Grant (LACSEG) (pdf) in recognition of the fact that as independent schools they no longer receive a number of services from local authorities (LAs), and must make appropriate provision for themselves or do without these services. LACSEG money is spent by local authorities on:

• Educational disadvantage: pupils with special needs but without statements, behaviour support services, educational welfare services, 14-16 practical programmes, assessing free school meals eligibility
• Educational enrichment: music services, visual and performing arts, outdoor education, museum and library services
• Risk sharing across schools: coverage of long-term sick, termination of employment costs, redundancy costs, capital asset management
• Shared administrative functions: school admissions, statutory and regulatory duties.

If financial considerations have indeed been a primary motivator for conversion, then is it also true that those with the greatest potential gains from conversion have done so in the greatest numbers? If this were true, then I suggest we should be able to identify likely converters using the following four criteria.
1.  Affluent schools within less affluent local authorities have most to gain from conversion
The share of LACSEG given to academies was calculated on a simple per pupil basis, yet a large proportion of LA services are used to support children with challenging educational needs. The smaller the local authority’s share of children from deprived backgrounds you take, the more you can benefit by taking your school’s LACSEG funding out of the central pot of money. Academy converters are indeed far more affluent (at under 9% free school meals eligibility for secondary schools) than the remaining state maintained sector (around 15% free school meals eligibility). It is also true that academy converters take a disproportionately lower share of the LA’s free school meals population.
2.  Potential benefits of academy conversion are highest in areas with a high LACSEG (as calculated by DfE)
Where a local authority’s LACSEG spending, as estimated by DfE on their website, is very high, local schools have the most to gain on a per pupil basis from academy conversion. But data actually show the local authorities that have lost the greatest numbers of schools to conversion are actually those with low LACSEG spending.
3.  Benefits should be highest in areas where the discrepancy between DfE-calculated LACSEG and ‘true’ LACSEG were highest
Many schools realised that they would receive a significant short-term windfall  from conversion because the DfE had incorrectly calculated the current expenditure on LACSEG services by local authorities. The discrepancy between calculated LACSEG and true LACSEG in an LA is anywhere between zero and almost £500 per pupil.* We might therefore expect that conversions were highest in areas where this discrepancy was particularly high. But the data show this isn’t so. In fact, some of the areas with the highest discrepancy in their calculation have had no secondary school convert to academy status (e.g. Islington, Barking and Dagenham, Hartlepool).
4.  Those in most financial need – with deficits or falling populations – should be most likely to convert
Some schools argued that they felt they had no choice but to convert to academy status to compensate for cuts in funding to their school. It is true that many schools have seen falling income year-on-year, but if this was an important motivator for conversion then those in the most financial need should have been most likely to convert. However, data show that those already in deficit in the 2009/10 financial year (the most recent for which we have all-England data) were no more likely to convert than others. Also, those with falling pupil populations were also no more likely to convert.
So, although those schools that have converted to academy status so far widely cite financial considerations, the characteristics of the converters are not as predicted if this were the sole consideration. What can we conclude from this? It is possible that many heads and governors do not have the right financial data to make the decisions about conversion, and had they done so, even greater numbers would have recognised the short-term windfall and so converted. Or alternatively political considerations for those sitting on governing bodies widely influenced the conversion decision, which is why more Conservative local authorities have seen the greatest rates of conversion. However, we should also not dismiss the idea that, at least in some local authorities, the sense of cohesion across local schools remains very high and that they value the quality of the services the local authority provides to help them provide schooling for children, particularly those in more challenging circumstances.
* Thanks to Chris Cook of the Financial Times for these estimates

Rebecca Allen

Chris Cook, the Financial Times education correspondent, has been writing about the Department for Education’s suggestion that the School Teachers’ Review Body (STRB)should consider whether greater variation in teachers’ regional pay is needed. He notes that greater variation in teacher pay would create a bizarre situation where schools in our most successful region (London) become even more generously funded, with a deterioration in funding in places where schools appear to struggle.
This observation raises the interesting question as to why London schools do so well, given that the high cost of living should make it difficult for them to recruit and retain the highest quality teachers. Why don’t the capital’s best teachers simply migrate to Stoke or Blackpool where their salary would provide them with a nice family home and a higher standard of living?
I would suggest that there are four possible explanations for this phenomenon, and it is not straightforward to decide which is the most important:
1. It’s all about the (extra) money.
London schools receive about £1,000 more per secondary school pupil but most of it is simply used to pay teachers their estimated £5,000 wage premium. This pay premium isn’t sufficient to cover greater housing and living costs in the capital. Once this greater wage bill is paid, London schools have very little left to spend on other resources that might contribute to their success.
2. Non-money inputs to London schools are of higher quality.
I would suggest that London schools have strikingly good governance because they are able to draw on a large pool of highly skilled, well educated people (many of whom do not send their children to local schools). They also benefit from some innovative sponsor input from the (original) academies programme and greatly draw on their own networks of expertise to share best practice through programmes such as London Challenge. However, I doubt this is sufficient to explain why London’s performance is so strong.
3. Teacher retention isn’t as important as we think.
Our stereotypical view of a London teacher’s career is that they work in London schools for four or five years, then move to another region when they find they cannot buy a house or support a family on a teacher salary in the capital. An argument could be made that this type of teacher drift doesn’t actually matter because teachers quickly attain their maximum effectiveness within a year or two of qualifying (there are several pieces of evidence on this from the US). The idea here is that a young and inexperienced teacher workforce can be dynamic, energetic and effective. However, as it happens, the age and turnover statistics for the London workforce aren’t quite as stark as you would guess:

• 25% of teachers in London schools are under 30, compared to 19% outside London
• 13% of teachers in London schools have a tenure at their school of less than one year, compared to 10% outside London
• Teacher turnover is about 21% in London, versus 19% nationally.

All this suggests that we need an explanation as to why London teachers are not leaving the capital in droves.
4.  London teachers are stuck in London!
The Department for Education (DfE) reports that geographical mobility is known to be lower for teaching than it is for other graduate professions. However, there hasn’t been a great deal of research done into why this is the case. One observation I would make is that 75% of teachers are women. Some of these women will be in dual earning households with a partner in a professional job. Where this is the case their household’s geographical mobility will be restricted by their capacity to both find jobs in a new region. There is every conceivable type of professional job available in London, but far fewer in a city like Stoke or Blackpool. This might explain why Stoke schools don’t have the pick the country’s greatest teachers, despite the low cost of living.
Rebecca’s references are DfE’s report to the STRB (pdf, 1mb) and DfE’s research report RR183 (pdf, 3mb). She is a joint author of the DfE’s research report.