John Jerrim.

The results from PISA 2018 are to be released tomorrow. And, as with every PISA release, there will be the usual media circus.

There will, of course, be all the obvious things to look out for – have PISA scores for England in reading/science/maths gone up or down, are any countries doing surprisingly well/badly etc.

But, as always with PISA, the headline results are not always as straightforward to interpret as on initial inspection.

The purpose of this blogpost is therefore to flag some important things to remember when the results are released.

John Jerrim.

PISA has now been running since 2000, with England participating in every cycle. Yet involvement does not come cheap. It costs more than £2m every three years for England, Wales and Northern Ireland to take part. Not to mention the burden it places upon nearly 500 schools.

It therefore seems important that we consider whether all this time and effort is worthwhile. Is England really getting enough out of its continued participation in the PISA study?

This blogpost focuses on four key reasons why England has participated in the PISA study, and the value that they bring.

John Jerrim.

When the PISA results are released every three years it is now little surprise when a set of east Asian nations (e.g. Singapore, Taiwan, Japan, South Korea) dominate the top spots in the rankings.

These nations typically substantially outperform most English-speaking western nations, with one important exception – Canada.

This is illustrated by the table below, which shows the top 20 countries in terms of average scores in reading, science and mathematics.

John Jerrim.

If anyone has ever read one of the international PISA reports or seen Andreas Schleicher present they will know that the OECD is rather fond of cross-national scatterplots. These illustrate the relationship between two variables measured at the country level.

Take, for instance, the chart below. This has been taken from one of Mr Schleicher’s blogposts, and illustrates the relationship between a country’s test scores and its rate of economic growth. It has been interpreted by the OECD as showing “that the quality of schooling in a country is a powerful predictor of the wealth that countries will produce in the long run”.

Source: Research by Hanushek and Woessmann, via the OECD Education and Skills Today blog

Sounds convincing, right?

The trouble is, correlation does not equal causation. And, despite the OECD’s obsession with such cross-country relationships, they can often deceive.

(more…)

John Jerrim.

Every time PISA results are released, concerns are raised about the methodology that underpins the work.

One area that has come in for repeated criticism is how the test scores of students are actually produced, as in this article, which asked whether PISA was “fundamentally flawed”.

Such concerns were exacerbated by a seminal paper by Svend Kreiner and Karl Bang Christensen who claimed that their results indicated that using PISA to compare countries was “meaningless”.

John Jerrim.

When the reform to GCSEs was initially announced, under the watch of Michael Gove in 2014, the intention was to link performance on the new GCSE exams to the PISA test.

Now, as far as I am aware, this link between PISA and national examination standards has not been established. Instead we have the comparable outcomes policy [PDF] and the national reference test to ensure standards are comparable over time.

Yet the interesting question remains – how do the ‘currency’ of GCSE grades and PISA test scores translate?

John Jerrim.

It is widely considered important that children read regularly. A wide range of previous research has linked reading during childhood to improved language skills and higher levels of academic achievement more generally.

But does it matter what children choose to read? Does flicking through a magazine or reading a newspaper have the same benefits for young people as becoming engrossed in a novel? A lot less evidence currently exists on this.

In a research paper published earlier this year, my co-author Gemma Moss and I decided to explore this topic in detail.

John Jerrim.

In England we are fortunate to have a lot of data available about school pupils and how they are achieving academically at school.

Organisations such as FFT aim to make this data available and easily digestible to schools through services such as Aspire so that it can be used to inform the decisions of teachers and school leaders.

But how does the way schools in England make use of data compare to schools in other countries?

John Jerrim.

Since 2009, parts of China have been participated in the OECD’s triennial PISA testing.

In 2009 and 2012, Shanghai topped the international rankings and by quite some distance, with 15-year-olds in this Chinese city estimated to be up to two and a half years ahead of their counterparts in England.

Yet China’s participation in PISA also led to controversy. (more…)

John Jerrim.

The PISA study is usually used to measure educational achievement across countries.

However, as well as taking a two hour test covering reading, science and mathematics, young people are also asked to complete a background questionnaire.

This covers a wide range of topics, such as self-confidence, life satisfaction, expectations for the future and out-of-school activities. There is hence a whole host of things – outside of educational achievement – that PISA allows us to compare across the world.