John Jerrim.

How much can we trust government reporting of key statistics? Not just the headline findings, but the basic details underpinning them? Those things that it’s important for consumers of data to know if they want to form their own independent judgement about the strength of the evidence available?

In my new paper released today, and forthcoming in the Review of Education, I report what I consider to be a worrying lack of transparency surrounding some aspects of the reporting of the PISA 2018 data for the UK.

This blog is the first in a series posted today looking into some of the (more…)

John Jerrim.

In my new paper released today, and forthcoming in the Review of Education, I report what I consider to be a worrying lack of transparency surrounding some aspects of the reporting of the PISA 2018 data for the UK.

This blog is the second in a series published today looking into some of the issues. Here I focus upon the non-response bias analysis conducted in England and Northern Ireland – but that didn’t get reported. (more…)

John Jerrim.

The second blog in this series discussed how a non-response bias analysis had to be undertaken for England’s and Northern Ireland’s PISA 2018 data.

The interpretation of the aforementioned bias analyses (by the OECD and the Department for Education) was that the PISA samples for these countries were ‘representative’ and ‘not biased’.

But is this really the case?

This blog presents evidence (more…)

John Jerrim.

With exams cancelled again in 2021, concerns have resurfaced over potential rampant “grade inflation”. Many saw this as a perennial problem throughout the nineties and noughties as GCSE and A-Level pass-marks went up year-upon-year. When the Conservatives came to power in 2010, it was something they vowed to end.

Indeed, some argued it was this obsession, over avoiding grade inflation and maintaining standards, that led to the disaster we saw with the awarding of examination grades last summer.

But how have GCSE standards really changed over time? This blog takes a look. (more…)

John Jerrim.

The results from PISA 2018 are to be released tomorrow. And, as with every PISA release, there will be the usual media circus.

There will, of course, be all the obvious things to look out for – have PISA scores for England in reading/science/maths gone up or down, are any countries doing surprisingly well/badly etc.

But, as always with PISA, the headline results are not always as straightforward to interpret as on initial inspection.

The purpose of this blogpost is therefore to flag some important things to remember when the results are released.

John Jerrim.

PISA has now been running since 2000, with England participating in every cycle. Yet involvement does not come cheap. It costs more than £2m every three years for England, Wales and Northern Ireland to take part. Not to mention the burden it places upon nearly 500 schools.

It therefore seems important that we consider whether all this time and effort is worthwhile. Is England really getting enough out of its continued participation in the PISA study?

This blogpost focuses on four key reasons why England has participated in the PISA study, and the value that they bring.

John Jerrim.

When the PISA results are released every three years it is now little surprise when a set of east Asian nations (e.g. Singapore, Taiwan, Japan, South Korea) dominate the top spots in the rankings.

These nations typically substantially outperform most English-speaking western nations, with one important exception – Canada.

This is illustrated by the table below, which shows the top 20 countries in terms of average scores in reading, science and mathematics.

John Jerrim.

If anyone has ever read one of the international PISA reports or seen Andreas Schleicher present they will know that the OECD is rather fond of cross-national scatterplots. These illustrate the relationship between two variables measured at the country level.

Take, for instance, the chart below. This has been taken from one of Mr Schleicher’s blogposts, and illustrates the relationship between a country’s test scores and its rate of economic growth. It has been interpreted by the OECD as showing “that the quality of schooling in a country is a powerful predictor of the wealth that countries will produce in the long run”.

Source: Research by Hanushek and Woessmann, via the OECD Education and Skills Today blog

Sounds convincing, right?

The trouble is, correlation does not equal causation. And, despite the OECD’s obsession with such cross-country relationships, they can often deceive.

(more…)

John Jerrim.

Every time PISA results are released, concerns are raised about the methodology that underpins the work.

One area that has come in for repeated criticism is how the test scores of students are actually produced, as in this article, which asked whether PISA was “fundamentally flawed”.

Such concerns were exacerbated by a seminal paper by Svend Kreiner and Karl Bang Christensen who claimed that their results indicated that using PISA to compare countries was “meaningless”.

John Jerrim.

When the reform to GCSEs was initially announced, under the watch of Michael Gove in 2014, the intention was to link performance on the new GCSE exams to the PISA test.

Now, as far as I am aware, this link between PISA and national examination standards has not been established. Instead we have the comparable outcomes policy [PDF] and the national reference test to ensure standards are comparable over time.

Yet the interesting question remains – how do the ‘currency’ of GCSE grades and PISA test scores translate?