The new Government is expected to make an announcement about the future of the census following a recent ONS consultation. In this blog based on his inaugural lecture at UCL, Professor Oliver Duke-Williams, Professor of Population Information and a senior member of the Centre for Longitudinal Study Information and User Support (CeLSIUS) team, asks how the past can inform that future.

The most recent census taken in this country was also perhaps the most unusual – the pandemic affected both the way in which people completed the form and the answers they gave. In addition, the vast majority of households completed the census online, where only a small proportion had done so in 2011, when the option was first available in this country. It may also be the last of its kind: alternative data sources offer the potential for different ways of counting the population, bringing with them their own opportunities and threats.

So why do we have a census, and for whose benefit is it? There are a number of purposes: censuses aim to count all people and households in the country, gathering a snapshot of the population at a particular time. They can also have a more subtle purpose: to assert the existence of a nation state, and to define the people that comprise the nation – both in terms of overall numbers of those included and perhaps excluded, and also in the categories and classifications chosen by the state to enumerate them. On a more general level, censuses in most countries including the UK are used for planning, in the hope that the state can make better decisions given better information.

Our census started in 1801 and the first few were very different to the ones we have now: they collected aggregate data rather than data on individuals, but from 1841 they began to become more robust, and primarily based – just as today – on individuals living in households.

We can use census data for many purposes. Whilst historic data is most commonly used to trace family histories, it can also be used more collectively via curated research datasets. For instance, data from Cornwall in 1891 shows the rapid rise in the number of girls named Florence after Florence Nightingale became famous. It is mostly a bit of fun, but allows use to think about the role of celebrity in popularising names, and using census data rather than birth records, also to think about naming practices amongst groups of people with differing characteristics.

Contemporary censuses

Moving to contemporary censuses, as well as the more familiar aggregate results that tell us information – often vital for planning – about how many people there are with particular characteristics in the country, or in specific locations, we can also study the way that individuals change over time. Linking data over time, the ONS Longitudinal Study – a one per cent sample of all census respondents from 1971 to the current day – can enable us to observe the changes in anonymised individuals during their lifetimes – where they live, what their occupations have been, how their households form and change and so on. Amongst other things, this can provide information on social mobility over lifetimes, and on occupational mortality over long periods.

The census isn’t perfect – the pandemic caused a glitch, for example, because the census in Scotland was delayed to 2022 while the censuses in the rest of the UK took place in 2021. So those who moved from England to Scotland between those two dates were counted twice, and those who moved the other way from Scotland to England weren’t counted at all. More generally, some people will be missing – whether by accident or design – in any census.

Some of the questions have become outdated – for example we saw a growth in working from home from 11 per cent to 31 per cent between 2011 and 2021, which has left questions about travel to work looking somewhat out of date. The assumption is still that people have one job and they go to it every day using the same mode of transport, but for many people that’s not the reality any more.
So, the census does make mistakes, and this brings us to its future.

Can AI help?

Perhaps we need to think about whether AI can help – it’s been suggested that natural language systems might help people to ask questions of the census in ways that are easier than the existing interfaces; there are a number of researchers working towards this, and it is fair to say that some people find existing tools hard to use. The answers that come back will be numbers, and it may be that AI could help to present those numbers in a more understandable form.

Just to see how well this would work, I asked the CoPilot, a Microsoft AI tool, to draw one hundred people, and to make them representative of the current UK population. All of the four responses I received had more than one hundred people in them – sometimes far more – and more importantly they weren’t very representative; there were few young people, few people of colour and almost no people with visible disabilities.

So if we think that AI can represent data for us and simplify data for us, we don’t yet have evidence that it can do a good job of it.

But administrative data does have advantages. It gives new sources of data for which some of the costs are already met- health service data or tax data, for instance, collected as part of the day to day operations of the relevant departments, and which can be re-used to create demographic data.

With these sources we don’t have to wait 10 years for the next set of data. But there are problems because at the moment we don’t have a single identifier which will enable us to link the records of individuals across different administrative datasets, so linkage becomes more difficult and expensive.

Big data

There’s also interest in using big data such as mobile phone data to gain insights into people’s lives. But again, there are issues with identifying those individuals and linking their data with other sources; phone data could potentially tell us a lot about movement patterns, but discerning movement by different groups of people would be less easy to achieve.

So, should we retain a census? It has had a 200-year run and not many administrative operations can beat that, so maybe its time has come?

The poor arguments for keeping it are that it’s what we’ve always done. Better ones include the fact that it’s hard to gather some types of information from other sources. The census provides us with a trusted framework for sampling, and once it’s gone we can’t easily recover it.

But there are also arguments for a hybrid approach – census and admin data are different, and they tell us different things. We’re at an interesting branching point, in a way a bit like we were back in 1841 when census data collection began to become more robust. What will the next census look like? The future is currently unknown.

Oliver Duke-Williams is Professor of Population Information in the Department of Information Studies at UCL, where he also leads MA/MSc programmes in Digital Humanities. He is the Census Service Director at the UK Data Service and is also a Senior Advisor to the Centre for Longitudinal Study.

by Nicola Shelton

Back in the late 1960s there was concern that policymakers had too little information about births and deaths: death certificates recorded only limited information and even the occupation of the deceased could be recorded inconsistently. Similarly it was impossible to use information from birth registrations to look at patterns of fertility – how were children spaced within families, for instance? And so the ONS Longitudinal Study was born.

The 1971 Census had recorded respondent’s date of birth – as opposed to age –  for the first time. And that allowed statisticians to record data on a one per cent sample of the population – all those born on four dates of birth which were, and remain, a closely-guarded secret.

The ONS Longitudinal Study now holds records for more than a million people, none of whom have any idea that they are a part of the study. It’s only possible to join through being born in the UK or through migrating into it, and it’s only possible to leave by dying or emigrating. The study also holds information for those living with its members, but it doesn’t follow them up in the same way from census to census.

Similarly, the information provided on birth certificates did not allow for studies of birth spacing. Although such data could be obtained from the General Household Survey (GHS), the total sample sizes were too small for detailed studies.

(more…)