Questions about PISA 2018, part 3: How representative is the data for England and Wales?

John Jerrim.

The second blog in this series discussed how a non-response bias analysis had to be undertaken for England’s and Northern Ireland’s PISA 2018 data.

The interpretation of the aforementioned bias analyses (by the OECD and the Department for Education) was that the PISA samples for these countries were ‘representative’ and ‘not biased’.

But is this really the case?

This blog presents evidence for England (where a bias analysis was undertaken) and Wales (where a bias analysis was not undertaken) to illustrate how such statements are doubtful.

Clear evidence of bias

Let’s start with Wales.

Through a freedom of information request, I have obtained the GCSE mathematics grade distribution of the Welsh PISA 2018 cohort (who took their GCSEs in 2019). In Table 1, I compare this to the GCSE mathematics grade distribution in Wales for all pupils, using official government records.

It becomes immediately clear that the Welsh PISA sample under-represents pupils who achieve lower GCSE grades.

For instance, according to official government data, 41% of young people in Wales failed to achieve a GCSE C grade in mathematics in the 2018/19 academic year. Yet, the analogous figure based upon PISA 2018 sample was just 31%.

Table 1. The GCSE mathematics grade distribution in Wales: PISA 2018 in comparison to the national distribution.

Notes: Figures for Wales refer to GCSE mathematics alone (not numeracy). For Wales, data on PISA grade distribution based on freedom of information request submitted by the author . Data on “official” grade distribution taken from here, using data for the 2018/19 academic year. Average PISA scores by grade based upon Gambhir, Dirie and Sizmur (2020: Table 3.3) – and refers to data on best GCSE grade out of numeracy and mathematics.

Table 2 below – using information I also obtained through a freedom of information request – presents the same evidence for England.

The same pattern holds; lower-achieving pupils tends to be underrepresented in the English PISA sample.

In other words, despite the claims made by the OECD and the Department for Education, there is at least some evidence that bias may have crept into the UK’s PISA sample.

Table 2. The GCSE mathematics grade distribution in England: PISA 2018 in comparison to the national distribution.

Notes: Green/red shading illustrates where the percentage achieving the grade is higher/lower in PISA than the national grade distribution. Figures for England based upon this data set and this data set.

 How might this affect PISA scores for England and Wales?

In my new paper published today and forthcoming in the Review of Education, I have tried to estimate the impact that the underrepresentation of low-achieving pupils may have had upon average PISA scores in England and Wales.

Full details about how I have tried to get a handle on this can be found within my paper. The headline finding, though, is that I estimate average PISA scores in England and Wales have been inflated by around 10 and 15 test points respectively, compared to if a truly representative sample had been drawn[1].

Table 3 then illustrates what this would mean in terms of England’s and Wales position in the PISA 2018 “rankings”.

The reported average PISA mathematics score for England was 504 points – significantly above the average across industrialised (OECD) countries. But, had a truly representative sample been drawn, I estimate England’s PISA mathematics score would have been around 494 – roughly the same as the OECD average.

Table 3. Average PISA mathematics scores in England and Wales – an illustration of the potential upward bias in the 2018 results.

Notes: “Reported” figures refer to the official PISA scores as reported in Table D1.1 of this England national report. “Adjusted” figures refer to my estimate of average PISA scores in England and Wales, after accounting for issues of non-participation.

A similar story holds for Wales. Their ‘official’ PISA mathematics score of 487 puts Wales in line with the OECD average (489). Yet, if a truly representative sample been drawn, I estimate their average score to be significantly lower (473 points).

This would put average mathematics scores of pupils in Wales in line with countries such as Malta and Belarus.

Now such thought experiments assume that there are not similar problems with the PISA data in other countries. This may not be true. Indeed, I have written previously about issues with the Canadian PISA data while there has also been recent controversy in Sweden due to the very high number of pupils it has excluded from PISA.

But two wrongs don’t make a right! And, as illustrated in the first blog of this series, there are clear reasons to suspect such problems may be more acute in some countries (e.g. the UK) than in others (e.g. Belarus, Finland and Singapore, where overall participation rates are over 90%).

[1] It has not been possible to produce equivalent figures for Scotland and Northern Ireland, due to a lack of information about the grades achieved in national exams (GCSEs in Northern Ireland or nationals in Scotland) for the PISA 2018 cohort.

More questions about PISA 2018

• Part 1: In Scotland, were key changes taken into account?
• Part 2: Did certain schools select out of the study in England and Northern Ireland?