What?

At ARC I think it would be fair to say we strive to develop and improve not only on an individual level, but also on a group level. One of the ways we are doing this is through our RSE (Research Software Engineer) Initiatives – aiming to advance/evolve the RSE team to improve collaboration and delivery of the best possible software. They involve taking a more objective look at the current processes within our department and determining, by consensus, whether some of these processes need to be updated, or if a new solution should be devised. These are not overnight quick-fixes but rather, slow & steady progressions in the right direction.

We’ve focussed on 3 main areas: Professional Development, Best Practices and Knowledge Sharing.

As we’ve recently reached the 6 month mark of embarking on this journey, I thought I’d share an overview of each initiative’s aim and how we’re doing.

Professional Development

Notable people: Connor Aird, Stef Piatek & soon to be Paul Smith

This is about understanding how we currently decide to upskill (soft and technical) ourselves, what opportunities there are and how we can enable and support more/better opportunities.

The way we decided to figure out what people are doing regarding their professional (and to some degree personal) development was by interviewing them.

At the time of writing almost all the interviews have been completed and data gathered, being prepared for analysis.

Best Practices

Notable people: Haroon Chughtai, Kimberly Meechan & Emily Dubrovska

This looks at how much we engage with establishing and following best practices with technologies, languages and tools. We also want to determine whether there are areas where we could formalise/document this for future RSEs – a notable example is within the Python Tooling Community.

We decided it would be worth modelling the approaches of the Python Tooling Community and seeing whether there are other language/technology communities within ARC that don’t have best practice guidance but would benefit from it. This was done through a survey.

At the time of writing, the next groups of interest are Web Development and DevOps – both in the stages of requirements gathering/gaining an idea of what guidance could be documented or be built on, as well as looking into how it could best be delivered. 

Knowledge Sharing

Notable people: George Svarovsky & Amanda Ho-Lyn

This is about understanding how we currently share knowledge across the group – particularly project information – and how we can improve our current systems to be more usable and make information more accessible.

We decided to do a survey to see how people felt about how information is currently shared and also how much they actually felt they knew about different aspects. There were also some mentions of discontent about where information was posted and shared across a plethora of platforms.

At the time of writing, we have added a mini landing page to the ARC GitHub (note that you must be part of the org to see it) in an attempt to centralise relevant links to various places – this is a living thing and can be updated as necessary. We have also sent out a survey (thank you to those who took the time to complete it) and have plans to act on the results – see my post with more details about this (coming soon).

Thanks to everyone who’s been a part of this and continues to help us improve – especially to Asif who is forging the way ahead. And keep an eye out for more surveys! 😁

Over the last 15 months there has been much debate, hype and concern relating to capabilities of tools and platforms leveraging Large Language Models (LLMs) and media generators. Broadly termed Generative AI. The predominant narrative in Higher Education has been around the perceived threat to academic integirty and associated value to degrees. As such a lot of focus and discussion has focused on taught students, assessment design and “AI-proof” assessment. This has been coupled with concerns relating to the inability to reliably detect generated content, and the disproportionate number of false positives related to non-native English speakers text submitted to various platforms.

However, despite the proliferation of Generative AI enabled research tools and platforms, numerous workshops offering increased research output productivity and publications asking authors to declare whether or not these tools were used in producing outputs there has been limited discussion with relation to staff and research integrity.

Coupled with the publication of initial findings from a study on staff use of these tools by Watermayer, Lanclos and Phipps that included use to complete “little things like health and safety stuff, or ethics, or summarizing reports” and potential safety risks from fine-tuning models as reported in the Stanford Univeristy published policy briefing Safety Risks from Customizing Foundation Models via Fine-Tuning a workshop focusing on the interplay of Generative AI and research integrity and ethics was proposed as an AIUK Fringe event.

Research Integrity in an AI-Enabled World took place on Monday 25th March 2024. The aim was to explore how we think Generative AI enabled tools and platforms, could and should impact on the research process, and what the integrity and ethics implication are. Eventually aim would be to produce a policy white paper.

The event was organised so that there was a series of thought provoking talks in the morning, followed by a world-cafe style session in the afternoon. The event was held under the Chatham House Rule to enable open and frank discussion of the topic and arising issues.

The first set of talks predominantly focused on ethical issues. There were discussions on authorship, and the nature of authorship where multiple actors are involved e.g. training data creators, platform developers and prompters.  Bias in image generation, reinforcing misconceptions and stereotypes. Culminating in a talk on the University of Salfords evolving approach to Generative AI and research ethics.

The second set of talks was focused on current capabilities, limitations and implications of using Generative AI enabled tools in the research pipeline, predomintly focusing on qualitative analysis. This session included a discussion around evidence synthesis and the need to find more efficient methods whilst maintaining reliability and a breadth of knowledge, and different approaches using “traditional” machine learning approaches versus use of large language models. Enhanced capabilities of Computer Aided Qualitative Data Analysis Systems and implications for methodological approaches were also introduced and discussed. The session concluded with a talk from Prof Jeremy Watson about the work currently being undertaken by the UK Committee on Research Integrity’s AI working group, of which he is member. Key themes currently under consideration by UKCORI are:

• Governance
• Roles and Responsibilities
• Skills and Training
• Public Understanding and Expectations
• Attribution and Ownership – IP, etc.
• Understanding Data Inputs and Models
• Need for Research in AI and Integrity

During the world-cafe session participants addressed the following questions:

• What do we mean by Research Integrity in an AI-Enabled Research Environment?
• Are there degrees of Research Integrity based on discipline and how embedded AI use is in the research process?
• What are the key ethical and legal considerations?

Including the following participant proposed questions:

• Generative AI is extremely good at in-filling uncertainty, where details of images become filled with bias. Should the responsibility of bias be equally on a prompter who enables this by omission?
• Recalibration of government and private funded RI in AI? Isn’t this the foundation of biases for RI?

Outputs from the world cafe session will be analysed over the next few weeks, and workshop participants were invited to contribute to the development of workshop outputs.

Key themes that emerged from the event include:

• Transparency
• Criticality
• Responsibility
• Fitness for purpose
• Data protection and privacy
• Digital divide – privilege and harms
• Training – education

The workshop was well received by participants, with the participants rate their overall experience of the event as 4.71 out of 5.

The speaker sessions were rated as very good by over 70% of participants. With the world cafe being mentioned as a highlight of the event.

As the proposer, organising and the host of the event I can’t help but still wonder:

• Can we ethically and with genuine integrity use tools which are fundamentally ethically flawed?
• Why are we accepting of these issues?
• How should we be pushing back?

I will leave you with these words from Arudhati Roy with which I opened the event: