Guest post by Andrew Gray, Bibliometrics Support Officer

Altmetrics are the concept of “alternative metrics” – measuring the impact of research beyond scholarly literature. This covers a wide range of different things, ranging from social media discussions (e.g. Twitter or Facebook), mainstream news reporting, and grey literature such as policy documents. Understanding how research is being reported and discussed in these can help give us a broader understanding of the impact and reach of papers that we don’t see from looking at traditional scholarly citations.

UCL has a subscription to Altmetric, the primary commercial database for this information. It covers a broad range of materials. We also subscribe to a second source, which focuses purely on policy documents – Overton and can be a helpful complement.

There are several ways in which looking at altmetrics can give us information that wouldn’t otherwise be available. For example, we can see how different audiences outwith academia are responding to research, and we can look at what they’re saying to get an idea of the kind of response.

Some of the altmetric indicators (particularly Mendeley bookmarks) seem to have a close correlation with subsequent citations and can give us an early view of what citation figures may be like six months to a year in future.

Lastly, tracing policy citations through Altmetric or Overton can effectively demonstrate the wider research impact, for example, for use in a funding report or application.

Looking at activity

So what data can we see? Altmetric provides an aggregated “score” for each paper, indicating an overall activity level. While this isn’t a very exact measure, it lets us identify papers with high and low activity levels.

Looking over the past few years at UCL, the most obvious thing is that discussion of research is dominated by COVID-19. It accounts for thirteen out of the fifteen most heavily discussed UCL papers overall – by comparison, were we to look at pure citation counts, COVID papers account for none of UCL’s top fifteen overall, and only perhaps four out of the top fifteen from the past few years. This very striking difference highlights how altmetrics and citations can show different things.

The colour swatches on each show how the activity is broken down. For example, in this paper, we can see that most of the activity is from X/Twitter (light blue), with smaller contributions from Facebook (dark blue), news media (red) and blogs (yellow). Clicking through will let us drill down to see all the activity details.

Diving into data – day by day

One thing that surprised us with Altmetric is the sheer volume of data that they make available. Reports of 100,000+ papers can be downloaded, including DOIs and PubMed IDs, making it easy to link data to other sources such as RPS and InCites. This lets us do some analyses that wouldn’t be possible in other sources – but do tell us something unexpected.

For example, it gives us the exact date papers were published. Looking at around 50,000 UCL papers published in 2020-22, we find that the response differs depending on the day of the week – papers on Wednesday and Thursday are above-average, and papers on Tuesdays are below average.

In part, it is because some of the most prestigious publications have fixed publication days – most Nature papers are released on Wednesdays, for example. These journals have a large share of high-impact papers and an excellent publicising system.

The weekends are interesting. Not many papers come out on the weekends, but the ones that do, have a noticeable citation/bookmarking penalty compared to weekday ones, suggesting they are less impactful on average. And they make much less of a stir in the news media – a weekend paper is less than half as likely to get news coverage as a weekday one.

But social media has a sharp difference – Sunday papers get significantly more Twitter activity than Saturday ones. An intriguing mystery!

Using Altmetric at UCL

Altmetric and Overton are both available to any user at UCL. You simply need to log in to Altmetric using a UCL email, which will set up your user account. For Overton, you can browse the data without an individual account or set up an account to save searches and other functionalities.

We have integrated Altmetric with RPS, the central UCL publications database. Every two to four weeks, every paper in RPS since 2013 is exported, tagged with the UCL author(s) and associated departments, and uploaded into Altmetric.

This means that we can use the Altmetric dashboard to dig down into UCL outputs in some detail – we can ask it questions like “news stories in the last month referring to a piece of research published by someone in Chemistry”. It is also possible to save and circulate reports from the dashboard – this report shows the top 20 papers from Chemistry in 2023 by Altmetric Activity.

Similar functionality is not yet available for Overton, but if you would like to search for papers from a specific department, we would recommend generating a list of DOIs from InCites (or even from Altmetric itself!) and importing those as an advanced search.

We will be running introductory training sessions for both Altmetric and Overton in the coming term – please contact bibliometrics@ucl.ac.uk if you would be interested in attending these or booking a 1:1 meeting to go through the services.