Chris Husbands 
No-one likes inspectors. The Daily Telegraph reports that TV licence inspectors are three times more likely to be attacked by angry householders than by angry dogs. In Montana, a furious meat processing company owner launched a physical assault on the food safety inspectors who had described his plant as “putrid”. RSPCA inspectors were assaulted almost 250 times in one calendar year. So Michael Wilshaw perhaps has some way to go as criticism appears to come not only from schoolteachers but – strenuously denied – unnamed briefers in the Department for Education, criticism which, he said, left him “spitting blood”.
School inspection in England has a long history. Her Majesty’s Inspectors were established in 1839, and the nineteenth century reports of inspectors remain as invaluable a source on nineteenth century education as the reports of factory inspectors on working conditions.  HMI developed world-renowned expertise in inspection, though their principal role was to provide information and advice to ministers: it was calculated in the 1980s that at the then current rate of progress, each school could expect to be inspected once every 250 years.
HMI was transformed in 1991. OFSTED was established. Every school was to be inspected on a four yearly cycle and – a critical development – the inspection handbook, which had hitherto been a closely guarded secret, was published as the framework for school inspection. Inspection arrangements, managed by OFSTED and overseen by HMI, were contracted out. The framework has been revised regularly since 1991, and the inspection cycle has been varied, but the principle remains the same: regular inspection based on published criteria. Other countries have also developed inspectorates, and Melanie Ehren from the IOE is leading a cross-national study.
There is little doubt that the twin measures of regular inspection and published criteria have exercised enormous influence on the system, and mostly for good. One of OFSTED’s early straplines was “improvement through inspection”, and the key idea of examining the performance of all schools on the same basis is one, albeit only one, of the measures which have helped to raise expectations of what is possible, of what schools can achieve.
The problems for OFSTED have often been not the inspection framework, nor the principle of judgements: all the research on education assessment and evaluation is clear that evaluative judgements based on public criteria matter. Instead, there have been concerns about variability in the quality of inspection teams, about the reliability of their judgements, about the interaction between a public inspection regime and an ever-tighter accountability framework, and the very serious challenges of sustaining improvement in the most challenging of schools: “improvement through inspection” is a good mantra, but has proved far more difficult to demonstrate in practice. Rob Coe from Durham University has identified the problems for OFSTED: inadequate training in classroom observation produces unreliable judgements about quality, and poor ability to interpret complex data makes it difficult for many teams to contextualise what they see. In a high stakes environment, these weaknesses have profound consequences.
In all this, the issue is, perhaps, less inspection than the weight which is hung on it:  as Melanie Ehren’s project is telling us, inspectorates can work in very different ways. As the reported disagreements between the Department for Education and first the Chief Inspector of Schools and now the Chair of OFSTED suggest, inspection is extremely important. It shapes the way governments, practitioners and the public think about the school system. There are some tough lessons from the history of inspection: there are always tensions between inspectors and policy makers; inspection judgements need to be nuanced as well as incisive; there are always limits to what inspection can do; inspectors stand in the perpetual militarized zone between those who would centralise education and those who would decentralise it. In practice, OFSTED really owns only one asset: its evidence base, still the most comprehensive and thorough evidence base on what happens in classrooms anywhere in the world. It is what makes OFSTED important and relevant, however uncomfortable its findings may sometimes be to read. The independence and integrity of the evidence base are of critical importance. It has been, and remains, a precious commodity in English education.

Chris Husbands
Singapore is one of Asia’s great success stories – transforming itself from a developing country to a modern industrial economy in not much more than one generation. As OECD observe, during the last decade, its education system has remained consistently at or near the top of most major world education ranking systems, its curriculum and assessment system prayed in aid of reforms elsewhere.
I was lucky:  I spent four days in Singapore as part of an international academic advisory group with colleagues from South Korea and Ontario, working with the Ministry of Education and National Institute of Education to look at the next phase of their education research and development strategy.
Singapore is beginning the third phase of an ambitious long-term programme of education research, and will spend something like $110m (about £50m – but in a school system just one tenth the size of England’s) over the next five years on how to scale up classroom and school interventions which work.
Increasingly, the Ministry is using rigorous research to understand what is going well and what is not: the 2005 TLLM [Teach Less, Learn More] initiative is not reckoned to have achieved its goal of aligning pedagogy with the demands of the emerging Singaporean economy. A fascinating piece of detailed research into pedagogy in Singaporean classrooms, conducted by David Hogan, has set out to explain why, and how to move beyond the performative pedagogy driven by a high stakes assessment regime.
In other areas, Singapore has used international research to move beyond stale debates: just as wars break out again in Anglophone countries between proponents of ‘knowledge’ and ‘skill’, Singapore has made extensive use of Marlene Scardamalia’s work on knowledge building to design continuing professional development for its teachers. Intensive work is being conducted by the Learning Sciences Laboratory on the use of computer games for learning, including the Statecraft X app for citizenship education. Manu Kapur’s work on ‘productive failure’ – essentially,  figuring things out with no scaffolds – has attracted the attention of  Time magazine.
Teacher education is critical. In the 1990s, after  a series of teacher shortages, Singapore began to evolve a strategy for teaching, putting in place conditions that raised the prestige of the teaching profession, providing new teachers with the best training, and putting in place extensive retention and professional development packages. Theory and practice are strongly linked in teacher education, and increasingly linked through the e-portfolio which all teachers use.
For all this, many of the pre-occupations of the Singaporean government and academics were curiously familiar: the performance of boys, the differential performance of children from different socio-economic backgrounds, the place of the arts and culture in education, the relationship between technology and effective teaching, the engagement of lower attaining pupils in the mathematics curriculum, the role of education in a culturally diverse society.
Others echoed education debates which rage back and forth in England, for example, the worry that highly competitive examinations drive teacher behaviours which do not promote good learning. One senior government official lamented that for all the high test scores, “we have a lot of studies in Singapore which show us that children can score highly on maths tests without understanding mathematics”. There is huge concern about the impact of a steeply hierarchic secondary school system on overall performance. For all the concern with citizenship education, the school system still operates under tight political constraint.
But what Singapore does have is a superbly designed delivery system for education. There is a clear strategic focus on policy from the ministry of education, which is staffed by ferociously able civil servants, many with PhDs, based on broad consent across politicians and professionals. The place of the National Institute of Education at Nanyang Technological University is key in translating policy into action and undertaking research which feeds back on the impact of policy. Whilst it can be seen as a monopoly provider of teacher education, leadership development and education research, the fact that NIE is the only provider gives it and the ministry a clear strategic role on leading the system.
But as one senior civil servant told me “even within a small system [just 30,000 teachers], it is not always a given that different sectors can work well together. It has taken us some years to build the structure and culture for things to come together.  Ultimately, people are the key”.

Paul Morris
The results of PISA 2012 will be announced tomorrow. Billed in the media as “education’s Olympics”, it compares the academic achievements of 15-year-old pupils around the world, ranking them according to performance and developing a compelling narrative of “winners” and “losers”.
From previous exercises, we can expect an epidemic of reform proposals as policy makers in England and those who seek to influence policy attempt to identify evidence as to: which countries are successful; what works elsewhere; what features can be borrowed to improve the lot of low performers. These questions will be pursued under the rationale of developing (or maintaining) ‘world class’ schools.
It will be portrayed as an objective/scientific exercise in evidence-based policy making, a straightforward case of learning from the best. Yet this portrayal has not, in contrast to other countries such as Germany, characterized the prevailing approach in this country to date. Rather, the PISA data, along with other international tests, have most often been selectively raided to defend and promote the preferred ideologies and preferences of those who make and seek to influence educational policies. The appropriate maxim is less “show me the evidence and I’ll decide policy” and more a case of “I know the right policy, I’ll find the evidence for it.”
I illustrate this point with two examples of how such tests have been appropriated to support existing views, opinions and prejudices about schooling, and add a note of caution.
The first example comes from the recent release of the Organisation for Economic Cooperation and Development’s Programme for the International Assessment of Adult Competencies (PIACC) on Oct 8. There was significant media coverage of the poor results recorded for the UK’s 16- to 25-year-olds and the following propositions were advanced by pundits to explain the poor results: young people give up mathematics and English too early; the absence of a performance-based pay system for teachers; the poor quality of teachers; the lack of discipline in schools; the dumbing-down of the curriculum; low investment in further education; the reduction of resources for young adults; and grade inflation.
Some of these claims were supported by reference to the opposite feature operating in a country that performed well. If however one ventures into the PIACC report none of the above school focussed “explanations” featured in the OECD’s attempt to explain the results in the UK. They identified social background (the children of parents with low levels of education have significantly lower proficiency than those whose parents have higher levels of education) and the nature of employment (low skilled jobs limit the development of skills) as the major impact on the levels of performance in UK.
The second case is the Coalition Government’s 2010 White Paper on Education, which was based on improving England’s position in international league tables by learning from high performers. As the Foreword explains:
… Alberta to Singapore, Finland to Hong Kong, Harlem to South Korea – have been our inspiration. 
Two main problems were identified: the quality of teachers and the lack of school autonomy. Where the White Paper argues the need for high quality teachers it relies heavily on a report by McKinsey & Company that cites as its evidence Finland, Singapore, South Korea and other high performing countries in the 2006 PISA exercise. However, when the McKinsey report later promotes the value of multiple and shortened routes for teacher education, the evidence to support this claim is not derived from any high performing country in PISA. Rather it uses evidence from Charter Schools in the USA and the Teach First scheme in England.
The White Paper uses the same strategy and promotes the need for multiple, shortened and school-based teacher education programmes, which are not a feature of the high performers. With regard to school autonomy, the White Paper bases its claim on the assertion that “everyone knows it is effective” studiously avoiding reference to the McKinsey report, which argued that there was no connection between school autonomy and pupil performance. Subsequently in 2010 the OECD argued that there is no simple relationship and that forms of autonomy designed to foster competition between schools, as is the case in England, did not improve results.

Peter Mortimore

I have been aware of problems in the education system for as long as I can remember.
As a child growing up in the late 1940s, I was persecuted by the fear of failing the 11+. Both my elder sisters had succeeded and my first brush with probability theory was recognising that similar success for me had to be questionable. And for 80 percent of each generation it was so, with many children bearing this hallmark of educational failure for the rest of their lives.
One of my jobs as a young teacher was in a school with 12 classes streamed on the basis of three timed tests. The validity and reliability of such assessment was highly questionable, yet it determined the course of future lives. One of my – very clever – pupils moved up a stream each year but, because he only stayed for the five years of compulsory secondary schooling, never even reached the upper half.
As a classroom researcher in the 1970s I sat through hours of lessons often in poorly designed, uncomfortable classrooms.  Some lessons were stimulating but others were desperately boring regurgitations of an uninspiring syllabus.
Today the education system, in many ways, is much improved.  Yet urban eleven-year-olds and their parents face a bewildering array of schools. City parents, in such a market economy, have to gamble on which preferences to express, taking into account their geographical location, the SATs results of their offspring, their religious (or non-religious) leanings and their willingness to pay for private tuition.
Academies – answerable only to the Secretary of State – founded with the noble aim of providing a better deal for poor children (just like the “public schools” of past centuries) are re-positioning themselves and using their generous resources to attract pupils with the best odds of success. Free “parent-led” schools are popping up even where there is ample provision and are frequently dedicated to particular faiths. Exams appear to be growing more difficult and universities are getting more expensive. Most sadly, English childhood appears to be much less happy than that of many of our neighbouring countries.
The English education system seems as far from a universal, inclusive, system as it ever was. Yet, the best teaching I have seen anywhere has often been in England and, in general, schools seem well-led. I believe these problems have been caused by recent governments’ (from all political parties) determination to turn the education system into a competitive market economy complete with league tables and punitive inspections.
In contrast, the Nordic school systems that I have observed operate in a different, happier, culture. They provide high quality pre-school provision and admit children into school one or even two years later than in England. The pace of learning is more relaxed and, wherever possible, failure is avoided.  Nordic young people continue developing and frequently overtake the ‘English early starters’ and become well-educated adults.
So what can we do to improve the system?  How can we learn from our North European neighbours? How can we draw on the strengths and mitigate the weaknesses of our current arrangements?  And how can we persuade our Government to create a fairer, happier and more effective education system better suited to life in the 21^st century?
My proposals are to try and ensure greater fairness in the system:
• Fairness in all funding: no more favoured schools getting large bonuses. The only payment above actual costs should be an extra sum for pupils with special needs.
• Fairness in governance – with all schools having uniform powers and working within the same national context.
• Fairness in what can be taught – with a limited National Curriculum available to all pupils.
• Fairness in licence to innovate in organisation and pedagogy.
• Fairness in inspections – with the aim being to ensure that all schools are above an acceptable level rather than attempting to fine grade them  on the basis of unreliable – and very limited – knowledge.
•  Fairness in assessment – helping as many pupils as possible reach the highest levels rather than seeking artificially to restrict success on the mistaken Kingsley Amis principle that “more will mean worse”.
•   Fairness in the allocation of pupils to schools so that all schools recruit a “balanced intake” – pupils who find learning easy and those who do not; those coming from relatively advantaged social, cultural and economic family backgrounds and their opposites.
Of course these are huge challenges. My suggestions require much elaboration and refinement. They will be resisted by the politicians associated with the current system and by parents satisfied with the privilege enjoyed by their children. Other people will have to be convinced that what is proposed will be better.
But such a set of ideas offer a chance of solving the worst problems and of creating a fairer education system better suited to life in the 21st century.
I’ll be talking about these ideas at a seminar at the IOE on 10 December. hope you will come along and contribute to the discussion.

Andy Green
This month’s OECD Survey of Adult Skills revealed, yet again, the shockingly low levels of basic skills of many young people in England compared with their counterparts in other countries. Like the International Adult Literacy Survey (IALS), conducted 16 years ago, it shows that England has an exceptionally large proportion of adults with very low levels of literacy and numeracy – the so-called long tail of under-achievement. But more disturbingly, the new report reveals that there has been little improvement amongst our 16 to 25-year-olds during a period when their peers in other countries have advanced rapidly.
Uniquely, in England there is very little difference in skill-levels between age groups, and yet inequality in adult skills, measured by the variance in the scores, is still worse than in almost every other country.
Politicians have responded by bickering about which administration is most to blame – the Coalition or New Labour. But, as the comparison with the IALS results shows, this is beside the point since national performance in the mid 1990s was also relatively poor. In fact the relative failure of mass education in England has a long history, implicating governments of all complexions, with deep political and cultural roots which are not amenable to quick policy fixes.
As I argued in my book Education and State Formation – first published in 1990 and now re-issued in an extended edition ­­– England was one of the last major powers to develop a national education system and the most reluctant to put it under state control. This relative underdevelopment in state education, which to informed 19^th century commentators seemed so anomalous in the world’s most industrialised nation, had a number of historical causes.
Whereas educational development in many advanced countries was driven by an intensive process of nation-building, in Britain the state was already largely consolidated, so there was little incentive to use public education for this purpose.  Successful early industrialisation, owing little to educational provision, taught the wrong lessons, and enabled deep complacency about the importance of skills to economic development. That this was a problem only became evident to the elites during the “second industrial revolution” in the late 19^th century, when Britain’s shortcomings in applied science and craft skills  became self-evidently a major barrier to industrial innovation and efficiency. It was the belated recognition of England’s relative skills deficits that finally spurred the construction of a public education system.
Underlying all other causes of England’s educational backwardness, however, was the pervasive culture of political and economic liberalism, with its veneration of free markets and hostility to the state. This deep-rooted individualist creed blocked the development of national education for many decades after it had become obvious that voluntary provision – by Churches and charities – could not meet the educational needs of the people or the economy.
It also left a potent legacy. The public education system finally put in place at the end of the 19^th century remained exceptionally fragmented and socially divided, with elite interests still dominating in the provision of secondary education – not least through private schooling – and with technical and vocational education still fatally undervalued.
These underlying flaws in the organisation of mass public education have never been rectified. Comprehensive education – introduced half-heartedly and with so much organisational variation that it never looked like an integrated national system – is now being dismantled. There was no national curriculum until 1988, more than a century and a half after most continental nations had one, and it has now been made optional for thousands of state-funded schools.
We are the only country in the world to persist in the absurdity of having national examinations organised by commercial organisations. And since the 1980s, all governments have been hell-bent on marketising education, creating, in the name of diversity and choice, a byzantine complexity of school types, and a school “system” so fragmented that it barely warrants the name. As a consequence, the idea of education as an essential public good is progressively undermined, and the inequalities which have always been the hallmark of English education become ever wider.
I concluded the first edition of my book with a clear warning to policy makers:
“If the past has any lessons at all it is that the mechanisms of the market and the ideology of laissez-faire serve education very ill indeed. It would be a sad irony if the country which was last to create a national education system, and which never quite completed the job, should be the first to dismantle it. It remains to be seen whether, in the name of market liberalism, England again becomes the ‘worst educated country in Europe.”
After three decades of reforms, which draw on a failed 19^th century liberal model of education for inspiration, we are now where we have always been, with an education system which serves the elites but manifestly fails to promote a high standard of education for all. With such policies our performance relative to other, rapidly improving, countries can only deteriorate, with England’s labour force starved of the skills to compete effectively in the global economy.

 ___________________________________________

Education and State Formation: Europe, East Asia and the USA offers an explanation of the long-run causes of England’s relative educational underdevelopment. In its new chapter on Education and State Formation in East Asia, it also shows how the newly industrialised countries of the East are using education to overtake us.

John Jerrim 

Last Friday the Sutton Trust published a very interesting report questioning the validity of global educational rankings. Having written extensively on this subject myself I can only welcome this report as making an important contribution to policymakers’ understanding of international comparisons of educational attainment. Yet the report also brought to mind the robustness of cross-national comparisons of another area of great policy interest – social mobility.
Readers have probably heard that social mobility is low in the UK by international standards. A number of sensationalist stories have led with headlines such as “Britain has worst social mobility in western world“ and that “UK has worse social mobility record than other developed countries“.
Leading policymakers have made similar statements. To quote England’s Secretary of State for Education Michael Gove: “Those who are born poor are more likely to stay poor and those who inherit privilege are more likely to pass on privilege in England than in any comparable country”.
But is this really the case? Are we sure social mobility is indeed lower in this country than our international competitors? Or is it the case that, just like global league tables of educational achievement, there remains great uncertainty (and misunderstanding) surrounding cross-national comparisons of social mobility?
The answer can actually be found by exploring a little further academic research that has been published on the Sutton Trust website. Figure 1 is taken from a Social Mobility Report published on 21 September 2012.
Figure 1: International comparisons of social mobility – Sutton Trust report 21^st September 2012

Saving the technical details for another time, the longer the bars in this graph, the less socially mobile a country is. Here we see a familiar story; Britain ties with Italy as being the least socially mobile.
Figure 2, however, tells a different story. This is taken from another report published by the Sutton Trust just three days later.
Figure 2: International comparisons of social mobility – Sutton Trust report 24^th September 2012

This graph plots a measure of income inequality (horizontal axis) against an economic measure of social mobility (vertical axis). Thus the closer a country is to the top of the graph, the lower its level of social mobility. Now, it appears that the UK may actually be more socially mobile than France, Italy and the US, and very similar to countries like Australia, Canada and Germany. Perhaps even more surprisingly, the UK is also similar to Sweden, Finland and Norway. Indeed, the only country that we can have any real confidence that the UK is significantly different to is Denmark.
Why is there such a contrast between these two sets of results? The trouble is, cross-national studies of social mobility have to rely upon data that are not really cross-nationally comparable. Rather, data of varying quality have been used in each of the different countries. Individuals are interviewed at different ages, using different questionnaires and survey procedures. Indeed, even different statistical analysis methods are used. No wonder, then, that social mobility in the UK can look very different, depending upon which dataset and method of analysis are used.
So although global rankings of educational attainment can be misleading, so can those of social mobility. In fact, problems with international comparisons of social mobility are often significantly worse. Yet this does not seem to stop journalists and policymakers making bold claims that “Britain has some of the lowest social mobility in the developed world“. Things are rarely so black or white in the social sciences – and social mobility is no exception. This uncertainty should be recognised when journalists and government officials report on social mobility rankings in the future. Otherwise, I fear for the credibility of this extremely important social issue.

Chris Husbands
Some weeks are just hectic. Four North American cities in four days. In Washington, New York, Boston and Toronto I led seminars, worked with school and university leaders and talked to policy-makers. The cliché, of course, is that England and North America are divided by a common language, and once I had translated my own ideas into American – “elementary”, not primary schools; “principals”, not headteachers; “school districts”, not local authorities (except, of course, that the roles and responsibilities are quite different) – there was a good deal of common ground.
As we await the latest National Curriculum proposals, American public schools grapple with the Common Core standards. While England’s Chancellor was announcing substantial changes to national pay agreements for teachers, Ontario teachers began a series of one day strikes in protest against their province’s plans to address its $14.4 billion deficit by saving money on teacher conditions of service.
So the questions I was asked, and the questions we discussed, had an easy familiarity:  securing system-wide change in ways which enhance both excellence and equity; funding public education systems in ways which do not disadvantage the least privileged; finding ways to move excellence practice around schools more quickly than teachers themselves can be moved; ensuring that the huge prizes of the digital revolution can be placed in the service of high quality learning rather than the other way round.
The past 20 years have seen a remarkable transformation in global education: the sorts of discussions I had with colleagues at George Washington University, Columbia Teachers’ College, Harvard and the Ontario Institute for Studies in Education (OISE) last week flow from global, not local, comparisons of education. We see it in England too: the facility with which school leaders and policy makers talk about the lessons of PISA, or the complexity of curriculum comparisons with the Pacific Rim. Education, and discussion of education policy, has gone global. Maybe that is no surprise in universities – used to thinking about international relationships since the Middle Ages – but it was striking that in Washington, in New York, at Harvard and at OISE, public (that is, of course, state) school teachers had joined the audiences and wanted to talk about their practice in the light of what I was saying about England.
And when they did so, of course, they threw up more of the complexities of education practices cross-culturally. Secondary teachers in England are not surprised to teach about 200 students every week, across the age and attainment range. High school teachers in Ontario teach closer to a hundred. In the United States, the figure is smaller still. That’s partly driven by different approaches to curriculum (for instance, American high school students take fewer subjects at a time, with more lessons a week for each) as much as it is by different conditions of service – but it makes a sharp difference to comparisons.
The relationship between quantitative and qualitative measures of school performance is not straightforward – in neither the United States nor Ontario is school review supplemented by an inspection model, so that school performance assessment in New York is considerably cruder than in England. And so on: the complexities of educational practice overlay differences in curriculum, funding and accountability.
But the thirst for comparison is real, and increasingly drives some very ambitious programmes of advanced professional learning. At Harvard, I met and worked with the team – and some of the students – on their new Harvard leadership programme. It is a three-year, full-time programme – yes – three years, full-time – for mid career professionals who have already been school leaders, seeking to prepare them for system leadership. The programme’s participants and the Harvard team leading it argued that the complexity of the challenges demand such a programme.
School systems are fascinatingly different: a shifting kaleidoscope, where the familiar can suddenly appear in a very different light and where apparently dissimilar features turn out to be quite familiar. On the Wednesday afternoon, walking to give a lecture at Columbia, I happened to be out on the street at the same time as school was ending. Knots of teenagers wielding mobile phones walked along the street laughing and joking. An hour later, when discussion turned to the challenges of making mobile phone bans in school stick, l felt on genuinely familiar ground.

 Chris Husbands
Two major studies of international attainment in education have been published: the four-yearly Trends in International Mathematics and Science Study (TIMSS) and the five-yearly Progress in International Reading Literacy Study (PIRLS). Both have been extensively reported and tell quite different things. On international comparisons, says the BBC, England “has slipped in science, but is top 10 for primary and secondary maths”.  The Daily Telegraph account, picking up on the science story, noted that England had fallen behind Slovakia and Hungary, whilst the BBC, looking at mathematics performance, declared that English pupils were amongst the best in the world.
Looking beyond the headlines, Chris Cook in the Financial Times  extracted data on the performance of Chinese heritage students in England from GCSE and, with the statistical wizardy of which he is capable, showed that such students in England perform almost as strongly as the students in high performing Pacific Rim systems.
International rankings of educational performance are now a routine feature of the education news cycle. The OECD PISA rankings come around every three years, on a different cycle from PIRLS and TIMSS.  My calculation is that edu-statistics geeks are going to have to wait until 2052 for the three to coincide – which makes 2052 like one of those rare astronomical events when several planets line up.
Of course, PISA,  PIRLS and TIMSS measure slightly different things at different ages – so the discrepancies between their methods, foci and findings prompt endless debate about their significance. The indefatigable North American blogger Yong Zhao has explored the complexities of what the figures show and suggested that the surface interpretation of PISA, TIMSS and PIRLS is almost certainly misleading. Each, moreover, is taken by a changing number of countries, so that arguments about whether performance rising or falling are endless.
But there is another, intriguing debate to be had about PISA, PIRLS and TIMSS. In this post  so far I have written about national performance. But all three studies draw on samples, and the sampling frame is different. So PISA compares, for example, Korea, which is a country,  with Hong Kong and Macau, which are special administrative regions, with Ontario, which is a  province in Canada. TIMSS includes the results for Massachusetts and Florida, which are not countries or provinces but States of the Union. This is a puzzle for comparison. We know, for example, in England, that schools in London out-perform schools nationally – though they have not always done so – and one report (again in the Daily Telegraph) suggested that the Department for Education in England is considering entering each English region separately in PISA 2015. But English regions are neither provinces, nor states, nor administrative regions, nor countries:  they are geographical agglomerations of local authorities.
In most descriptions of  international performance, the comparisons are said to be not between countries but between “jurisdictions”.  But educational jurisdictions are rarely defined, and sloppy comparisons follow between quite different entities. An educational jurisdiction must be a unit which shares some characteristics: for starters, common curriculum and assessment standards, teacher recruitment, development and remuneration policies and overall system development policies. It would make little sense to break a jurisdiction (for example, England) which shares these characteristics into smaller units – although that is exactly what PISA does in publishing results for Shanghai rather than for China.
It makes sense to treat Hong Kong separately from Shanghai: they are very different education systems with different histories, linguistic traditions and educational structures. It probably makes sense to deal with Ontario differently from Alberta in PISA: in Canada education is strong a provincial responsibility. American states are a different matter, however: as increasing federal funds are allocated to States dependent on conformity with US-wide educational requirements, the responsibilities are shifting. But even national comparisons are fraught with difficulty: is it reasonable to compare New Zealand (population of 4 million) with Japan (population of 120 million)? The conventional argument is that Asian jurisdictions – with often highly centralised education systems in cultures which place a high premium on education – do exceptionally well in mathematics assessment. A cheekier reading of PISA might suggest that smaller jurisdictions do better than larger jurisdictions.
There is a vast amount to be learnt from international comparisons of educational performance, and many countries have reported “PISA shock” when results suggest hitherto unexpected weaknesses. But it’s worth remembering that the unit of comparison is often complex. International comparison can be full of pitfalls – and one of these is “what is a jurisdiction”?