Back at the end of April, UCL hosted its first-ever Replication Games thanks to a grant from the Research Culture Seed Fund awarded to Kirsty Wallis, Head of Research Liaison at UCL Library Services, and Sandy Schumann, Lecturer in Security and Crime Science and local network lead for the UK Reproducibility Network. The event was facilitated by Derek Mikola from the Institute for Replication (I4R).
Three teams worked together to try to reproduce the analysis and results of a published paper – all in one day. They used only the information provided by the authors’ descriptions of the methodology and analytical approach, as well as public datasets and, to the extent that this was available, code. The groups consisted of a mix of researchers at different career levels and from altogether nine different disciplines.
The day kicked off early with an introduction by Derek, who got everyone set up in their groups, well provisioned with coffee and snacks. The groups had already been working together in the lead up to the event itself, selecting a paper from a list compiled by the I4R and familiarising themselves with the materials the authors had shared. Throughout the day, the teams encountered several challenges. Some discovered that packages that were used to run the analysis were no longer supported, meaning, ultimately, that the core analysis could not be conducted. Others identified questionable procedures to re-code variables, which meant that once variables were treated appropriately, results were not reproduced. Only one of the teams came to the conclusion that the published findings were largely reproducible.
Overall, the replication games were a roaring success. Participants were able to connect with researchers from other disciplines and gain hands-on experience in the workflow of reproducing published work. They also reported that they developed a better understanding of how to present their own work in a manner such that it is detailed enough to be reproduced.
All teams will document their analyses in reports that will be published by I4R soon, so watch this space and we will share the final product!
The UCL Office for Open Science and Scholarship invites you to contribute to the open science and scholarship movement. Stay connected for updates, events, and opportunities. Follow us on Bluesky, and join our mailing list to be part of the conversation!
It has been a couple of weeks since our debut collaboration with our friends at LSE and the Francis Crick Institute and I can safely say that the festival was a roaring success. We all would like to extend a huge thank you to everyone that came to any of the events, in person or online, it was great to see so many people engaging with Open Science!
In case you missed it, the festival ran from 2 – 6 June and included an exciting array of sessions including creative workshops, informal networking, case studies, online and in person panel discussions and technology demonstrations. The full programme is still available online, or keep scrolling for links, recordings and upcoming content!
This session of making, sharing and storytelling has its own blog post – read it now!
Tuesday 3 June
Co-producing research with Special Collections: Prejudice and Power case study
UCL Special Collections presented their experiences of using co-creation to engage with rare book and archive collections especially as applied to the recent Prejudice in Power project, that consisted of a range of co-creation, community and academic initiatives that focussed on our holdings to respond to the university’s historic role in promoting eugenics. It also briefly discussed wider co-creation activity in UCL Special Collections, the lessons learned and how these are being embedded in practice.
Diamond open access (OA) is championed as a more open, equitable and inclusive, community-driven journal publishing model, especially when compared against other commercially owned, author pay and subscription models. Demand is rapidly growing but there is a lack of capacity and funding for journals to sustainably meet it. There are many barriers to solving these complex challenges, but one new initiative called the Open Journals Collective aims to disrupt the current landscape by offering a more equitable, sustainable and alternative solution to the traditional and established payment structures.
During this interactive session we heard from the conveners of the collective to learn about why and how it came about, what it offers and why it is needed. We also heard about the experiences with various OA journal models, as well as perspectives from a journal Editor who resigned from a subscription journal and successfully launched a new and competing diamond open access journal.
Professionalising data, software, and infrastructure support to transform open science
This workshop focused on the needs of both researchers and technical support, seeking to understand the answers to some fundamental questions: If you are a researcher – what do you need in terms of technical support and services? If you are a research technology professional – what skill and training do you need to be able to offer this support?
The team in ARC behind this fascinating session have shared a write-up about it which you can read on their blog page.
Wednesday 4 June:
Should reproducibility be the aim for open qualitative research? Researchers’ perspectives
Reproducibility is often touted among quantitative researchers as a necessary step to make studies rigorous. To determine reproducibility, whether the same analyses of the same data produce the same results, the raw data and code must be accessible to other researchers. Qualitative researchers have also begun to consider making their data open. However, for researchers in fields where cultural knowledge plays a key role in the analysis of qualitative data, openness of such data may invite misrepresentation by re-use of the data by researchers unfamiliar with the cultural and social context in which it was produced.
This event asked whether reproducibility should be the aim for open qualitative data, and if not, why should researchers make their qualitative data open and what are the other methods used to establish rigour and integrity in research?
How open is possible, how closed is necessary? Navigating data sharing whilst working with personal data
In the interests of transparency and research integrity, researchers are encouraged to open up more of their research process, including sharing data. However, for researchers working with personal data, including interview and medical data, there are important considerations for sharing. This event brought together researchers from a range of disciplines to share their experiences and strategies for open research when working with personal data.
Thursday 5 June: Open Research in the Age of Populism
Political shifts around the world, from the Trump administration in the US to Orban’s government in Hungary, are making it more important than ever to have reliable research freely available. However, these governments are also making it more risky to openly share the results of research in many countries and disciplines. Alongside the political censorship of research in some countries there are also changes to research funding, research being misrepresented and used to spread misinformation online, and concerns about the stability of open research infrastructure which is funded by the state. In this session the panellists considered the value of open knowledge, the responsibilities of individual researchers and institutions to be open and how you can protect yourself when making your research openly available.
With the rapid growth of AI tools over the past three years, there has been a corresponding rise in the number of academics and students using them in their own writing. While it is generally agreed that we still expect people to be the “authors” of their work, deciding how to interpret that is often a nuanced and subjective decision by the writer. This in-depth panel discussed how we think about “authorship” for AI-assisted writing.
This session was so in-depth that the panel and the chair have worked together to create a summary of the discussion, complete with the resources and themes shared, which you can read on a separate blog post.
Registrations are now open for UCL’s first Replication Games, organised by the Office for Open Science & Scholarship and UCL’s UKRN local network chapter. The event will be run by the Institute for Replication (I4R), and it is supported by a Research Culture Seed Grant.
The Replication Games is a one-day event that brings together researchers to collaborate on reproducing and replicating papers published in highly regarded journals. Researchers participating in the Replication Games will join a small team of 3-5 members with similar research interests. Teams verify the reproducibility of a paper using its replication package. They may conduct sensitivity analysis, employing different procedures than the original investigators. Teams may also recode the study using the raw or intermediate data or implement novel analyses with new data. More information can be found on I4R’s Website.
Teams will be guided in all activities by Derek Mikola, an experienced facilitator from the I4R. After the event, teams are encouraged to document their work in a report that will be published on the website of the I4R. Participants are also eligible to be granted co-authorship in a meta-paper that combines a large number of replications.
This event takes place in person. Lunch and afternoon snacks are provided.
Who are we inviting to register?
Registration is on a ‘first come, first serve’ basis. We invite MRes students, doctoral students and researchers, post-docs, and faculty members at UCL to apply. Although students and scholars from all disciplines can apply, we hope to attract especially those working in the social sciences and humanities.
Participants must be confident using at least one of the following: R, Python, Stata, or Matlab.
Close
