Close
Evidence-based practice: why number-crunching tells only part of the story
By Blog Editor, IOE Digital, on 14 March 2013
Rebecca Allen
As a quantitative researcher in education I am delighted that Ben Goldacre – whose report Building Evidence into Education was published today – has lent his very public voice to the call for greater use of randomised controlled trials (RCTs) to inform educational policy-making and teaching practice.
I admit that I am a direct beneficiary of this groundswell of support. I am part of an international team running a large RCT to study motivation and engagement in 16-year-old students, funded by the Education Endowment Foundation. And we are at the design stage for a new RCT testing a programme to improve secondary school departmental practice.
The research design in each of these studies will give us a high degree of confidence in the policy recommendations we are able to make.
Government funding for RCTs is very welcome, but with all this support why is there a need for Goldacre to say anything at all about educational research? One hope is that teachers hear and respond to his call for a culture shift, recognise that “we don’t know what’s best here” and embrace the idea of taking part in this research (and indeed suggest teaching programmes themselves).
It is very time-consuming and expensive to get schools to take part in RCTs, (because most say no). Drop-out of schools during trials can be high, especially where the school has been randomised into an intervention they would rather not have, and it is difficult to get the data we need to measure the impact of the intervention on time..
However, RCTs cannot sit in a research vacuum.
Ben Goldacre does recognise that different methods are useful for answering different questions, so a conversation needs to be started about where the balance of research funding for different types of educational research best lies.
It is important that RCTs sit alongside a large and active body of qualitative and quantitative educational research. One reason is that those designing RCTs have to design a “treatment” – this is the policy or programme that is being tested to see if it works. This design has to come from somewhere, since without infinite time and money we cannot simply draw up a list of all possible interventions and then test them one by one. To produce our best guess of what works we may use surveys, interviews and observational visits that took place as part of a qualitative evaluation of a similar policy in the past. We also used descriptions collected by ethnographers (researchers who are “people watchers”). And of course we draw on existing quantitative data, such as exam results.
All of this qualitative and quantitative research is expensive to carry out, but without it we would have a poorly designed treatment with little likelihood of any impact on teacher practice. We may find that, without the experience of other research, we might carry out the programme we are testing poorly for reasons we failed to anticipate.
The social science model of research is not ‘what works?’ but rather ‘what works for whom and under what conditions?’
Education and medicine do indeed have some similarities, but the social context in which a child learns shapes outcomes far more than it does the response of a body to a new drug. RCTs may tell us something about what works for the schools involved in the experiment, but less about what might work in other social contexts with different types of teachers and children. Researchers call this the problem of external validity. Our current RCT will tell us something about appropriate pupil motivation and engagement interventions for 16-year-old teenagers in relatively deprived schools, but little that is useful for understanding 10-year-old children or indeed 16-year-olds in grammar schools or in Italian schools.
The challenge of external validity cannot be underestimated in educational settings. RCTs cannot give us THE answer; they give us AN answer. And its validity declines as we try to implement the policy in different settings and over different time frames. This actually poses something of a challenge to the current model of recruiting schools to RCTs, where many have used “convenient” samples, such as a group of schools in an academy chain who are committed to carrying out educational research. This may provide valuable information to the chain about best practice for its own schools, but cannot tell us how the same intervention would work across the whole country.
Social contexts change faster than evolution changes our bodies. Whilst I would guess that taking a paracetamol will still relieve a headache in 50 years’ time, I suspect that the best intervention to improve pupil motivation and engagement will look very different to those we are testing in an RCT today. This means that our knowledge base of “what works” in education will always decay and we will have to constantly find new research money to watch how policies evolve as contexts change and to re-test old programmes in new social settings.
24 Responses to “Evidence-based practice: why number-crunching tells only part of the story”
- 1
- 2
-
3
Building evidence into education « The Learning Spy wrote on 14 March 2013:
[…] here’s an alternative view on the same event: Evidenced Based Practice: why number-crunching tells only part of the story by […]
-
4
Andrew Fowler wrote on 14 March 2013:
Rebecca is right to highlight the question of validity. In education, understanding the context is extremely important, which limits the wider applicability of many studies. Another objection to RCTs is an ethical one: whilst it may be defensible to give one or other treatment (or no treatment) to a sick patient (after all, they were sick already, one might argue), how ethical is it to apply the same level of experimentation on children? John Hattie’s meta-analysis of various educational interventions showed – in terms of months’ gain in schooling – the effect of a large number of possible ‘treatments’. It would be possible to construct an argument to carry out an RCT of some of the more effective strategies, perhaps, but it would be very hard to defend the testing of previously-untested or less successful strategies. Thirdly, I wonder whether there isn’t a fundamental difference between doctors and teachers in their attitudes to their ‘patients’ – at the basic level, I wonder how successful many teachers would be as impartial researchers in RCTs. A clinical approach belongs, surely, in the clinic and not in the classroom.
-
5
Evidence-based practice: why number-crunching tells only part of the story | Seyi's thoughts on education wrote on 15 March 2013:
[…] Evidence-based practice: why number-crunching tells only part of the story. […]
-
6
Charlie Owen wrote on 15 March 2013:
Whilst I very much welcome Ben Goldacre’s challenge to examine the evidence supporting all aspects of education, I think he fails to give sufficient attention to the differences between social and medical interventions. Rebecca Allen is right to draw attention to the external validity of RCTs in education – that is the extent to which it is possible to extrapolate from a trial to what would happen beyond the trial. An important difference between social and medical interventions is that you cannot disguise who is getting the new treatment in a social intervention: it is possible to get doctors to prescribe treatment without them or their patients knowing who is getting the new pill and who is getting the standard treatment. This is called a double-blind, and its importance in clinical trials is well established.
To take one of Ben Goldacre’s examples, if a new method of teaching reading were tested using an RCT it would be impossible to keep from the teachers – and possibly form the pupils – who was getting the exciting new method and who was getting the boring old method. And that is part of the problem: in studies of interventions the Hawthorn effect is well known – that is the effect that simply knowing you are taking part in a study makes people behave differently. Teachers will know if they have been assigned to the group teaching with the new method. Even though they have been chosen at random, they will feel especially chosen, and that it is they who are under test to deliver this new method, and not just the method itself. It would be natural to assume that this new method must be better, otherwise it wouldn’t be offered to pupils: if the method does not prove to be better, then it may them as teachers who have failed, and not the method. They will therefore put in more effort than if they had assigned to the control condition.
Worse than that, people are very nice: they know that the researchers have put a lot of time and effort into developing this new method of teaching reading, and their future careers depend on a positive outcome. Under those circumstances, what teacher is not going to put in that extra bit of effort to see that the researchers achieve their goal?
This inability to disguise from the participants which condition they are in – intervention of control – can fatally undermine many attempts at RCTs in education. Just knowing they have been chosen to deliver the new method will make teachers behave differently. If the new method does prove superior, and it is rolled out to all schools, the teachers being required to implement it will no longer feel special, pioneers of educational innovation. The performance of the new method might then fall to that of the standard method – or even below. In this way, the transparency of the intervention to those delivering it can compromise the external validity of RCTs in education. -
7
Ben Waugh (@benwaughuk) wrote on 21 May 2013:
So we need a better placebo for the control group: a shiny new method that is not expected to lead to any improvement in learning. That sounds even harder to sell than being part of a control group using the old method. Perhaps we need to compare two (or more) new methods in each trial, so we get a better idea of which if either leads to a greater improvement. We still don’t get a valid comparison with the old method this way though. On the other hand, if existing practice varies between schools it might be possible to find classes where some “old way” is a change from their own current system.
-
8
How can we tell snake oil from science? | Pragmatic Education wrote on 16 March 2013:
[…] The reaction from the educational blogosphere was cautious. Andrew Old questioned which problems randomised trials would solve, given that the problem is disagreement over educational aims. Tom Bennett thought it directionally right, but was sceptical of its practicality. David Didau said many thought it utterly impractical. Dr Becky Alan welcomed it, but urged us to ask not just what works, but what works under which conditions? […]
-
9
Neil Holden wrote on 16 March 2013:
I enjoyed reading your article, Rebecca. I’ve written a reflection having read Goldacre’s and your article on my own blog (at holdwise.blogspot.com). I’d quite like to respond to some of the comments here too. A lot of valid problems are raised but I think in many cases it is possible to overcome them.
Rebecca and Andrew mention the external validity problem, which I think is valid, but is often used to dismiss RCTs very quickly; how much does the way children learn e.g. basic numeracy really change with time and culture? I would want to see evidence that this is the case.
Andrew, you mentioned the difference between doctors and teachers, and how teachers would have to be impartial RCT researchers. Charlie also mentions that you cannot disguise who is getting an intervention and who is not. To overcome these problems, we’d need the large scale cooperation that Goldacre suggests. To borrow Charlie’s example, if you have a number of schools participating in a trial for a new reading intervention, the students aren’t going to be as aware that some are getting ‘the new and improved’ program and others aren’t as they would be if it was in the same school (or classroom!) You would just give half of your schools the new intervention, and half would do what they were doing before (which you would need to ensure is broadly consistent across the control group.) In many cases, the students wouldn’t have to know that they are receiving anything new or exciting, as it’ll be the first time they’ve done a particular course/unit.
Still, the teacher would know that they are doing the new and exciting method, and Hattie’s “Visible Teaching” says that for this reason we should look for an effect of a certain magnitude that goes beyond what would be expected from just the effect of trying something new. (The assumption being, as Charlie points out, that a teacher trying something new and exciting will put more effort into it and then do it better, often resulting in improved outcomes regardless of the intervention.)
Of course this would require massive cooperation and coordination, but it’s not theoretically impossible to do. I’d be interested to read the Cochrane autobiography to see if people thought the same about RCTs in medicine, but with enough determination and cooperation it was seen that these seemingly insurmountable practical problems can be overcome. -
10
Evidence Again | Sam Shepherd's Blog wrote on 20 March 2013:
[…] The main paper is published here, but with a summary as a Guardian article here and a response, of sorts, published by those fine folks at the Institute of Education can be found here. […]
-
11
Evidence-based practice: why number-crunching tells only part of the story | British Politics and Policy at LSE wrote on 23 March 2013:
[…] Government funding for RCTs is very welcome, but with all this support why is there a need for Goldacre to say anything at all about educational research? One hope is that teachers hear and respond to his call for a culture shift, recognise that “we don’t know what’s best here” and embrace the idea of taking part in this research (and indeed suggest teaching programmes themselves). […]
-
12
Eddie Carron wrote on 23 March 2013:
Evidence-based practice in boosting literacy skills.
The harsh reality that there are at least seven million illiterate adults in the UK is proof positive, if such were needed, that classroom practice is determined more by ‘fad and fashion’ than empiricism. The fact that ‘phonics’ has particularly strong ‘back to basics’ flavour with it’s inevitably powerful political appeal, means that science and commonsense have to take a back seat. I have no doubt that evidence and logic could quickly solve the conundrum of why, in one of the most expensive education systems in the world, almost a fifth of children do not even learn how to read and write confidently.
It was long thought that the 2% of children who arrive at school as competent readers were just a particularly gifted minority but in reality almost all children, given appropriate opportunity, could learn to read by following texts visually as they are read to by their mothers. The small number of sight words effortlessly internalised in this way by the brains’ pattern-seeking imperative, enables them to read other unfamiliar but similarly constructed words, extending sight vocabulary and conferring the ability to read without phonics instruction. The teaching of reading is a field that is constantly being over-grazed by a herd of ‘literacy experts’, who successfully manage to ignore the reality that we all learned our first language perceptually rather than by ritual instruction and that perceptual learning is the teaching strategy preferred by all successful commercial language teaching organisations such as Rosetta Stone.
A number of school groups are presently involved in the development of a perceptual learning strategy to restore literacy skills deficits. One of these, a bilingual school in Wales, discovered not only that their English but also their Welsh language skills improved dramatically. One Scottish school which has been using the approach successfully for a number of years has produced a video of its work at http://youtu.be/d-wlbCFVzto
If nothing else, their experience demonstrates how important it is that teachers keep their minds open to new ideas and maintain a healthy scepticism for the kinds of rampant dogma that seems to replace commonsense when governments intrude into the nation’s classrooms. Science which really only means ‘truth’ will eventually prevail over ignorance but the in the meantime, the price which is being paid by society at large and the children whose lives we are playing with, is a very heavy one. - 13
-
14
The Evidence-Based Educational Research debate. Information in Schools wrote on 7 April 2013:
[…] 14th March: Evidence-based practice: why number-crunching tells only part of the story. Rebecca Allen, Institute of Education blog. http://ioelondonblog.wordpress.com/2013/03/14/evidence-based-practice-why-number-crunching-tells-onl… […]
-
15
Why Learning Styles are important | Sam Shepherd's Blog wrote on 18 April 2013:
[…] a response, of sorts, published by those fine folks at the Institute of Education can be found here. There have been a number of other responses as well, all of them interesting: from the Guardian […]
-
16
Ben Goldacre’s Bad Education? – Guest Post By Jonny Scaramanga wrote on 11 May 2013:
[…] and they enjoy a great deal more variation than pancreases. As Rebecca Allen argues over at the IOE blog, social science research is far more context dependent than medicine. What works for my kid might […]
-
17
Sue Cowley wrote on 21 May 2013:
I missed this at the time you wrote it, but wrote something similar here:
http://suecowley.wordpress.com/2013/03/19/everything-in-moderation/
In teaching, context means so much. Plus the effect that an individual teacher can have, for good or for bad, is huge.
I really enjoyed reading all the comments as well, particularly what Eddie said here about reading/phonics.
Thanks. -
18
JFB wrote on 21 May 2013:
Both Rebecca’s Allens and Geoff Whitty’s comments to Ben Goldacres’ DfE Analytical Review seem to be saying ‘well of course we need randomised controlled studies in education, I have been saying this all along, however, I have always advocating going much further and using much more sophisticated and nuanced interpretation that applies only to education to say what works for whom and under what conditions’. This humbug is beyond tolerance on several levels, as firstly during Geoff’s 10 years premiership of the leading education research institution his organisation did not carry out any randomised control trials. Similarly, throughout Rebeccas’ career, apparently dedicated to this paradigm shift she has not previously carried one out. So this adherence to RCT seems a rather rapid conversion suspiciously coincidental with recent rise in criticism of the methods that come from outside education research establishment they are central members of. Secondly, their understanding of RCT methodology suggests the topic is ‘Google new’ to them as they say its’ different in education than it is in medicine because unlike medicine in education we must know what works for whom and under what conditions. When in medicine it is also necessary to know what works whom and under what conditions for example a drug supposed to treat Parkinson’s disease (what) should help those with Parkinson’s disease (whom) at this dose at this stage of the disease (what conditions). It is hard to imagine what experiments they are thinking of that don’t specify what, whom in what conditions. Moreover these comments seem to attempt to claim credit for inventing afresh the idea of what’s called the ‘methods’ section in the write up of any experiment in any research in any field and taught in the first week of any research methods course (apparently apart from education). Their hubris is further compounded by their attempt to imply that they have added substantially to the debate by suggesting that readers of education research need to be careful to bear in mind that studies on for example, 15 year olds Maths in average UK schools, would only apply only to 15 year olds Maths in average UK schools and not 4 year olds music lessons in a Viennese conservatoires. These comments seem to suggest that the Rebecca and Geoff think that teachers will not be able to make this inference.
Further clarifying the recency of their thoughts on the subject they seem to think it is informative to point out that experiments without theory aren’t very helpful, when any view of the history of experimental science suggests fairly clearly that doing experiments on things you think might work is more helpful that doing experiments on things you don’t think might work. They go on to suggest a solution to this problem is to ask for expensive qual studies with RCT, a call likely to put off funders rather than encourage them at a time when as Rebecca rightly points out schools and funders are million miles away from buying into the need for RCT. A step likely to preserve the perennial problem stymieing education research where whenever asked to research anything education researchers respond ‘oh its very complicated’, interpreted by funders to mean expensive and producing inconclusive results. All this seems to suggest Rebecca and Geoff may have gathered their thoughts on the issue of promoting RCT rather more hurriedly than they may have us believe. -
19
Blog Editor wrote on 23 May 2013:
JFB – a quick search of the IOE’s website will show that IOE colleagues have long been closely engaged in furthering quantitative research in education and specifically in understanding the role of (and conducting) RCTs in the fields of education and related areas of social policy. As Geoff notes in his blog, Ann Oakley made a similar case for RCTs over a decade ago, but at that time “faced a lack of funding on anything like the scale needed to realise her dream”. Funding will necessarily shape the kinds of research that is conducted. I’d also point you in the direction of Geoff Whitty’s 2005 BERA Presidential Address, which set out his position on the need for both quantitative and qualitative research in education, and the other factors that influence policy decisions, as reflected in his recent blog piece. Ben Goldacre’s report clarifies the issues regarding the need to know what works, for whom, under different conditions.
-
20
JFB wrote on 23 May 2013:
I think your points confirm mine quite well because the sum total of IOEs’ long standing engagement and furthering, the directors influence and support for voices in favour has produced no controlled experiments, indicating that its commitment is evidently fairly slight.
Also as IOE represents a large chunk the education research community its collective views have a great influence on funding decisions, therefore to say its hamstrung by funding decisions is contradictory as collectively it has a large say over those funding decisions. The brick wall Ann Oakley encountered, the historical and on-going absence of funding for RCT therefore is a good indicator of the lack of support from the field which senior IOE members are a major part of and all of a sudden seem to now to be trying to claim they have been fighting for all along.
It does not seem clear why Ann Oakley required prohibitive amounts of funding to carry out controlled experiments as essentially they only involve identifying a group of interest, selecting a representative number of them at random, giving them the intervention and comparing their outcomes with others who didn’t receive it. This is largely to same procedure as delivering an intervention under any circumstances only its’ necessary to record who has received the programme and who hasn’t. I have not needed extra funding in my experience of doing several RCTs in education. This suggests Ann’s dreams may have been quite elaborate.
Whilst I take your point that IOE education researchers have been contemplating the issue of experimental methods for a long time I am not clear how highlighting the length of their meditations addresses doubts about the depth and robustness of thought that has lead them to be halted in their progress by a consensus that unlike almost all other fields of enquiry educations’ complexity prevents them overcoming the challenge of comparing the outcomes for those receiving their practise to the outcomes of those experiencing otherwise the same circumstances but not receiving it.
I am afraid, I am not clear how redirecting me back to the report I am commenting on deals with concerns about how informed researchers are who feel that education faces complexities that exceed those in fields such as psychology, neuroscience, and physiology that overcome the complexities they face to almost exclusively use experimental methods to make progress. Surely justifying the position that education faces uniquely complex conditions rendering it impermeable to experimental methods involves showing what conditions in education are unique to it, especially problematic to control for and more complex than those overcome for example in psychology and behavioural neuroscience. -
21
Think Like An Education Secretary: Gove’s 2013 Reading List « Laura McInerney wrote on 24 December 2013:
[…] Reaction was fierce. Some loved the idea. (To be fair, it was a well-written report). Others were more sceptical. But either way it gave Gove great headlines and enabled him to use the phrase “enemies of […]
-
22
Modern Governor Policy initiatives vs. evidence in education – Modern Governor wrote on 4 September 2014:
[…] work saying that it welcomed the report and another response from the Institute of Education emphasised the need for qualitative as well as quantitative research in education. In this light, a new policy which […]
-
23
Stuff we like: A catch up | David and Torben for D&T wrote on 22 June 2015:
[…] the use of these kinds of large scale randomised control trials that have transformed medicine. Not all education researchers are convinced that these ideas will transfer to education as well as Goldacre claims, but this book is a good place to read Goldacre’s argument […]
-
24
"Marxists, Feminists & The Blob": Rethinking Gove's Outburst – LKMco wrote on 26 August 2015:
[…] rather than being falsely led to believe that some methods are all good and some all bad. Plus, as Dr Becky Allen summed up in a blog for the IOE, random control trials will be more effective when supplemented with research showing what works […]
I couldn’t agree more Rebecca and think that much depends on our ability to infuse the teaching profession with a sense of inquisitiveness about its work and a consequent interest in enquiry as a feature of professional development.