Close
Why does Vietnam do so well in PISA? An example of why naïve interpretation of international rankings is such a bad idea
By Blog Editor, IOE Digital, on 19 July 2017
John Jerrim.
When the PISA 2015 results were released in December last year, Vietnam was one of the countries that stood out as doing remarkably well. In particular, Vietnam was ranked 8th out of all the participating countries in science, with an average score of 525 test points. This was significantly higher than the average score for the United Kingdom (509), which was positioned 15th in the PISA science rankings.
This is not the first time that Vietnam has apparently excelled in PISA, with a strong performance from this country in the last round, conducted in 2012. Indeed, OECD Director for Education and Skills Andreas Schleicher wrote a whole article for the BBC, discussing a variety of reasons for this developing country’s stunning success.
But does Vietnam’s amazing performance in PISA, given that it is still a low-income developing country, mean we should rush to copy what they are doing in their schools (much like what the Department for Education has naively done in its attempts to copy Shanghai)?
No! Because Vietnam’s PISA results (and the league tables so beloved by policymakers) are giving us an inflated perspective on how well this country is doing in educating its young people.
To understand why, we first need to recall what the PISA study is trying to do. It is attempting to measure the reading, science and mathematics skills of the in-school population of 15-year-olds across the world once every three years. The key words in the sentence above are the in-school population. What about young people who are not in school, or have already left the education system completely? Simple – they are excluded from the group PISA is attempting to measure.
To put this problem into context, lets say PISA was a study of 17-year-olds rather than 15-year-olds. How would the UK do? My guess is pretty well, because many of our lowest-achieving pupils leave the education system at age 16 – and hence would be excluded from the study. Suddenly, the UK would have higher average scores and less educational inequality than many other countries across the world.
This is exactly what happens with the PISA results for Vietnam. According to the OECD’s own figures, only 48.5% of Vietnam’s 15-year-olds are actually included in the PISA study (see Table A2.1 of the report linked to here). Those that have been excluded (e.g. children who have left school early) are likely to be academically weaker than those who have actually been tested. Thus Vietnam’s PISA scores are artificially inflated, making this country’s education system to appear to be much stronger than it really is.
Can we get a handle upon how much impact this is likely to have had? If one digs through the many hundreds of OECD PISA tables, you can find an alternative set of PISA results where it is assumed that, in each country, 15-year-olds who are not included in the study all perform below the national average. The particular results I am interested in here are available in Annex B1.2: Table I.2.4d. Using this information, I can compare these alternative results to the headline PISA findings, and can consider the difference.
This is exactly what I do in Figure 1 below, with results referring to the 75th percentile of PISA science scores. (I focus on the 75th percentile as for Vietnam alternative results for the median do not exist – given that more than half of 15-year-olds do not take part). The horizontal axis presents the headline figures from PISA, while the vertical axis provides the alternative results after all 15-year-olds in the country have been included (assuming those who are not in school perform below the national median).
Figure 1. The performance of Vietnam in PISA 2015, before and after the inclusion of 15-year-olds who are not in school
Vietnam has been circled and, as we can see, is a major outlier. Specifically the ‘real’ performance of Vietnam is probably between 50 to 60 points lower than reported in the headline PISA rankings. Infact, in these alternative results, Vietnam is well behind the UK (the 75th percentile is 566 for the UK versus 519 for Vietnam), with Vietnam now ranked a lowly 47th in these revised rankings (with the UK sitting in 17th position).
It is critical to note that this result is not about an issue with sampling for PISA in Vietnam. Rather it is a limitation with how the study population is being defined. Either way Vietnam serves as an important case study of how hypnotic international rankings like PISA can be – and just how easily they can lead us astray. Much deeper and more considered interpretation of the results (and analysis of the data) is needed for PISA and other international studies to really be useful for education policymaking. At the moment, there continues to be far too much hysteria surrounding what often turns out to be some quite flaky results.
This article was first published in Research Intelligence, a termly magazine sponsored by the British Educational Research Association.
5 Responses to “Why does Vietnam do so well in PISA? An example of why naïve interpretation of international rankings is such a bad idea”
- 1
-
2
hai wrote on 29 May 2019:
In latest 10 years , Vietnam was ranked 7 times higher than UK in International Olympiad of math (IMO) http://imo-official.org/results.aspx.
- 3
-
4
Emeritus Profesor Rosemary Davis wrote on 19 July 2017:
You are, of course, right. I simply pointed up some of the obvious ones and should have mentioned that the variables that I identified were in Secondary school classrooms where conditions tend to be better than Primary. would be good to hear your suggestions for other factors.
-
5
PISA Rankings Are Misleading Because of Differences in Student Coverage – SOS Australia wrote on 20 July 2019:
[…] John Jerrim, an academic at the Institute of Education at the University College of London, has suggested that Vietnam’s PISA science scores are artificially inflated because those who were not covered by the sample (eg, left school before turning 15) are likely to be academically weaker than those who were tested. He assesses the impact of the low coverage of Vietnam’s 15-year-old students by comparing its science results with other PISA data where it is assumed that 15-year-olds who are not in school perform at or below the national median. He estimates that the “real” performance of Vietnam is probably 50 to 60 points below its mean science score. This would put Vietnam’s ranking in science at 35-40th, a far cry from 8th. […]
How wise to point up the differences in population being sampled. I have no idea how many people have actually gone into schools in Vietnam, let alone rural ones. I have, from north of Hanoi to the tip of the Mekong Delta. Conditions were similar to any other developing country – lack of desks and books, huge class sizes and teachers with limited training. No doubt these conditions have vastly improved but the real lesson is that unless one compares like with like, rather than ‘apples’ with’oranges’, the results will still represent ‘garbage in and garbage out’.