Before we compare mathematics, reading or science, here's some geography
By Blog Editor, IOE Digital, on 13 December 2012
Chris Husbands
Two major studies of international attainment in education have been published: the four-yearly Trends in International Mathematics and Science Study (TIMSS) and the five-yearly Progress in International Reading Literacy Study (PIRLS). Both have been extensively reported and tell quite different things. On international comparisons, says the BBC, England “has slipped in science, but is top 10 for primary and secondary maths”. The Daily Telegraph account, picking up on the science story, noted that England had fallen behind Slovakia and Hungary, whilst the BBC, looking at mathematics performance, declared that English pupils were amongst the best in the world.
Looking beyond the headlines, Chris Cook in the Financial Times extracted data on the performance of Chinese heritage students in England from GCSE and, with the statistical wizardy of which he is capable, showed that such students in England perform almost as strongly as the students in high performing Pacific Rim systems.
International rankings of educational performance are now a routine feature of the education news cycle. The OECD PISA rankings come around every three years, on a different cycle from PIRLS and TIMSS. My calculation is that edu-statistics geeks are going to have to wait until 2052 for the three to coincide – which makes 2052 like one of those rare astronomical events when several planets line up.
Of course, PISA, PIRLS and TIMSS measure slightly different things at different ages – so the discrepancies between their methods, foci and findings prompt endless debate about their significance. The indefatigable North American blogger Yong Zhao has explored the complexities of what the figures show and suggested that the surface interpretation of PISA, TIMSS and PIRLS is almost certainly misleading. Each, moreover, is taken by a changing number of countries, so that arguments about whether performance rising or falling are endless.
But there is another, intriguing debate to be had about PISA, PIRLS and TIMSS. In this post so far I have written about national performance. But all three studies draw on samples, and the sampling frame is different. So PISA compares, for example, Korea, which is a country, with Hong Kong and Macau, which are special administrative regions, with Ontario, which is a province in Canada. TIMSS includes the results for Massachusetts and Florida, which are not countries or provinces but States of the Union. This is a puzzle for comparison. We know, for example, in England, that schools in London out-perform schools nationally – though they have not always done so – and one report (again in the Daily Telegraph) suggested that the Department for Education in England is considering entering each English region separately in PISA 2015. But English regions are neither provinces, nor states, nor administrative regions, nor countries: they are geographical agglomerations of local authorities.
In most descriptions of international performance, the comparisons are said to be not between countries but between “jurisdictions”. But educational jurisdictions are rarely defined, and sloppy comparisons follow between quite different entities. An educational jurisdiction must be a unit which shares some characteristics: for starters, common curriculum and assessment standards, teacher recruitment, development and remuneration policies and overall system development policies. It would make little sense to break a jurisdiction (for example, England) which shares these characteristics into smaller units – although that is exactly what PISA does in publishing results for Shanghai rather than for China.
It makes sense to treat Hong Kong separately from Shanghai: they are very different education systems with different histories, linguistic traditions and educational structures. It probably makes sense to deal with Ontario differently from Alberta in PISA: in Canada education is strong a provincial responsibility. American states are a different matter, however: as increasing federal funds are allocated to States dependent on conformity with US-wide educational requirements, the responsibilities are shifting. But even national comparisons are fraught with difficulty: is it reasonable to compare New Zealand (population of 4 million) with Japan (population of 120 million)? The conventional argument is that Asian jurisdictions – with often highly centralised education systems in cultures which place a high premium on education – do exceptionally well in mathematics assessment. A cheekier reading of PISA might suggest that smaller jurisdictions do better than larger jurisdictions.
There is a vast amount to be learnt from international comparisons of educational performance, and many countries have reported “PISA shock” when results suggest hitherto unexpected weaknesses. But it’s worth remembering that the unit of comparison is often complex. International comparison can be full of pitfalls – and one of these is “what is a jurisdiction”?
3 Responses to “Before we compare mathematics, reading or science, here's some geography”
- 1
-
2
John Oversby wrote on 13 December 2012:
While these are significant comments they place less emphasis on the relatively small sample of knowledge assessed, on the challenges of ensuring that the tests are equivalent in different languages (we know they are not), the lack of assessment of major aspects of science and maths such as investigations, problem-solving, many aspects of affect such as perseverance and resilience, and the backwash effect of the assessments that has resulted in whole jurisdictions (!) focusing mainly on the assessable in these surveys in their teaching and monitoring programmes. There is a great danger of valuing the assessable, and not assessing the valuable, driven by the market focus on metrics. It is, obviously, an easy way out for politicians, who wish to avoid the difficult choices of characterising non-metric outcomes.
-
3
Alex Jones (@alexcj) wrote on 13 December 2012:
There is another problem with all these tests and that is system goals. PISA, TIMMS and PIRLS test different types of knowledge and understanding. The tests assume goals for education by assessing the selected knowledge. My greatest concern is that by focusing on these tests as summative judgements on the quality of UK education we are in danger of adopting the goals of an external body without reflection. There’s a poverty of philosophy in state education. It was a salient criticism of of the recent national curriculum documents that there was no statement of overall aims. What goals do we have as a nation for our education system? If those goals align with PISA, or TIMMS or PIRLS then fine – lets take very careful note of what they have to say.
The size and type of education jurisdiction is clearly an issue to consider in determining the nature of the education system within it. I first came across this when running the Royal Society’s project on the assessment of 14-19 science learning with King’s College London which reported in 2004, the same year as Tomlinson (http://royalsociety.org/uploadedFiles/Royal_Society_Content/policy/publications/2004/4294969761.pdf). Roger Brown gave a thoughtful presentation on the Australian state-based system which at the time had a variety of approaches to assessment, including Queensland which had no formal external exams and teacher moderation aided by a reliability test. Of course there is variety within the UK itself, with NI doing much better in primary schools now according to the latest comparisons.Wales is rethinking the size and role of its LAs in education. London is considering whether to have its own additional curriculum about one of the world’s greatest cities. Perhaps other conurbations will follow in the Shanghai, Singapore and Hong Kong model? There is no magic formula to decide on an ideal size, even within New York City where school districts are reshaped geometrically to follow local population changes and street patterns.