In November I had the opportunity to attend the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025) in Suzhou, China, an ancient water town about an hour from Shanghai. With its beautiful canals, classical gardens (including the famous Humble Administrator’s garden), and historic architecture, Suzhou provided a lovely backdrop to this year’s conference. I particularly enjoyed visiting Hanshan temple and walking through Pingjiang road and Shantang street with their narrow streets that run parallel to the waterways.

At the conference I presented our work “Machine-generated text detection prevents language model collapse.” This research, conducted with my PhD supervisors Professor Emine Yilmaz and Dr. Vasileios Lampos, addresses a growing concern in the era of large language models: as AI-generated content becomes increasingly prevalent online, future models risk being trained on an unknown portion of synthetically generated data. This creates a feedback loop that can result in model collapse, a degenerative process where models gradually lose linguistic diversity and degrade in performance over successive generations.

Our work first explored how different model decoding strategies affect the severity of collapse, revealing that certain sampling methods accelerate degradation more than others. Building on this, we introduced a prevention method that uses a machine-generated text detector to estimate the likelihood that each training sample is synthetic. We then apply detector-guided resampling to up sample high-confidence human samples and down sample likely AI-generated text. The results were encouraging, not only does this approach prevent model collapse, but it can also lead to improved performance compared to training on human data alone. This underscores the benefit of using synthetic data in model training, an area that is gaining lots of popularity in this field, but also the importance of data curation.

The conference offered a rich program of talks, tutorials, and workshops, exposing me to many new ideas and research directions I hadn’t previously encountered such as pixel language modelling and novel Mixture of Experts architectures. I particularly enjoyed the Allen AI keynote on the Olmo 3 model series and appreciate how transparent they are in their foundational model development. I also really valued the opportunity to connect with other researchers exploring similar questions around synthetic data, model collapse, and continual  learning, areas that are central to both my PhD work and my role as Chief AI Officer at Locai Labs.

At Locai, I’m applying these concepts to develop large language models from the UK without the extensive cost of training models from scratch by building on existing open-source models through continual learning. This work directly extends the themes in our EMNLP paper, as we explore how models can avoid catastrophic forgetting, the tendency to lose performance when trained on new data, by training on a carefully curated portion of their own generated outputs. This helps mitigate excessive distribution shift and hence forgetting during further fine-tuning. We have applied these methods to develop our first model, Locai L1 Large, which is available at locai.chat. I wrote about the model release in a technical blog, with the full paper coming soon!

I’m grateful to the UKRI Foundational AI CDT for supporting my attendance of EMNLP. Conferences like this are invaluable for connecting research with real-world impact, and I’m already looking forward to the next conferences!

Poster Session

Last month, students, researchers, and industry leaders gathered for an inspiring three-day CDT Showcase to explore cutting-edge developments in artificial intelligence, machine learning, and their applications across society. The event combined academic talks, poster sessions, interactive activities, and social events, creating a vibrant environment for learning and collaboration.

The Summer School opened with a lively icebreaker and networking session, setting the tone for three days of exchange and exploration.

Prof Marc Deisenroth

A thought-provoking discussion chaired by Prof Marc Deisenroth examined AI for Environment and Sustainability. Following a presentation by Marc, delegates were divided into groups and tasked with exploring and reporting back on topics, including ‘Where do you see potential for AI transformation in environmental modelling’ and ‘Where do you see potential for AI transformation in sustainable energy’. This was a great opening session, encouraging participants to reflect on the role of technology in tackling global challenges!

Later, keynote speaker Dave Palfrey, Chief Scientist at MindMage gave a fascinating talk on code modularity and division of labour,  drawing parallels between human collaboration and the way we design AI systems. The final talk was CDT researcher David Chanin’s work on improving sparse autoencoders.

Dave Palfrey

The day concluded with a poster exhibition showcasing the breadth of student research, before delegates headed to an escape room social event in central London – a perfect chance to test teamwork skills outside of the showcase!

UCL East’s Intelligent Robotics Lab

The final day featured an exclusive tour of UCL East’s Intelligent Robotics Lab, where our guests learned how AI and robotics are being applied to fields such as autonomous vehicles, sustainable agriculture, and industrial inspection.

The afternoon’s talks tackled major questions around the future of AI research. Highlights included Reuben Adams on AI safety and the automation of AI research, Laura Ruis on how large language models learn reasoning skills, and

Prof. Tim Rocktäschel

Prof. Tim Rocktäschel with his ICLR keynote on open-endedness and the automation of innovation. These sessions pushed participants to think critically about the pace of AI development and its long-term implications.

Looking Ahead

The 2025 CDT Showcase was a testament to the energy, creativity, and vision of the AI research community. From student-led breakthroughs to expert insights on sustainability, safety, and innovation, the event highlighted both the opportunities and responsibilities that come with shaping the future of AI.

We look forward to continuing these conversations and seeing where the next year of research will take us.

Best Poster and Programmed Art Prizewinners

1. Introduction:

The “Conference on Mathematics of Wave Phenomena 2025” was held at Karlsruhe Institute of Technology (KIT) in Karlsruhe, Baden-W”{u}rttemberg, Germany — about an hour away from Frankfurt by train — between the 24th and 28th of February.
Karlsruhe is an old and wonderful town dating back to the early 18th century, nestled between the Black Forest and the Rhine river, and was built by the old Margrave of Baden-Durlach, Karl-Wilhelm.
In fact, the city’s layout radially extends from the Margrave’s palace with all the streets converging at the palace, and it is where Karlsruhe gets its nickname — the fan city.
KIT is the largest research centre in Germany and was founded in 1815, and has a long list of renowned researchers who have worked there including Fritz Haber and Heinrich Hertz. The conference was chaired by Roland Schnaubel — an expert in hyperbolic and parabolic PDEs and spectral theory — and focused on developments in mathematical modelling, simulation and analysis of wave-type equations and phenomena; key fields in analysis and numerical analysis.

2. My Contribution

I was invited by an ex-UCL colleague to present my unpublished work on Integrating Traditional Numerical Analysis Solvers with Learnable Operators, in a minisymposium titled, “Intersection of Biomedical Acoustics and Machine Learning”, which was chaired by my secondary supervisor Prof. Ben Cox.
Unfortunately, I fell under the weather on the day of my Talk but I was able to present my work and it was well received by the full room I was presenting to, indicating that the focus of my work fit well within the minisymposium.
Despite being the only person at the conference researching the budding field of Neural Operators, there was a good amount of overlap of concepts and techniques with my fellow researchers in the minisymposium.
The talks by Andreas Hauptmann (Fast Fourier models in circular geometry for learned reconstructions in photoacoustic tomography) and Janek Gr”{o}hl (Tackling the photoacoustic inverse problems with semi-supervised deep learning) were of particular note as they focused on leveraging Machine Learning to accelerate or supplement traditional numerical solvers — a key area of research for me — and illuminated a new perspective on how to use ML to complement the strength of traditional techniques.

3. Conference Highlights

Thankfully, I was only under the weather for a day and so I managed to attend and listen to many fascinating talks — even if many of them required more than the 4 years of mathematics I received during my MSci.
Some of the most Interesting talks were focused on Photoacoustic Tomography (PAT) and Reconstruction, which is an area in biomedical imaging where a pulse of light causes tissue to heat up and emit wideband Ultrasound.
PAT (and PAT reconstruction) problems are difficult to model in the forward direction (from light pulse to Ultrasound detection) and even more so in the inverse/reconstruction direction (from received Ultrasound to material properties of the tissue) due to both: the need to model the photoacoustic effect (generating sound from light in a medium), and the Ultrasound transmission and scattering problems.
This is further compounded in the inverse/reconstruction direction due to compounding error from the noisy Ultrasound transducers, causing the reconstructions to be both expensive and error artifact prone.

The notable talks on this topic at Waves 2025 were:

“Self Supervised Sparce-Data Image Reconstruction for Photoacoustics” by Markus Haltmeier -This Talk focused on how to simplify PAT reconstruction by using an new sensor set up to translate a 3D problem into a 2D one, and by using ML models they developed called ‘Sparce2Inverse’ and ‘Noise2Inverse’ to either clean up the noisy data before using a classical reconstruction pipeline or to use traditional denoising techniques and then apply these models to clean up artifacts during reconstruction.

“Fast Fourier models in circular geometry for learned reconstructions in photoacoustic tomography” by Andreas Hauptmann et al. — This Talk focused on the computational difficulties which data-driven end-to-end models face and discussed how they have been building more efficient forward and adjoint models with the aim of bringing the computational cost of PAT reconstruction to be small enough for common medical use.

“Tackling the photoacoustic inverse problems with semi-supervised deep learning” by Janek Gr”{o}hl et al. — This talk focused on using Generative Adversarial Networks (GANs) in order to improve the forward model for PAT problems and to aim to make the results of digital twin experiments (computer modelling of the forward PAT problem in a real world example) look more like real experimental data.

“Physics-Informed deep learning for ultrasonic imaging” by Felix Lucka — This Talk focused on concepts close to my own projects, namely incorporating traditional Algorithms formulated on wave-physics (such as delay-and-sum or f-k migration) as differentiable layers in a ML Architecture frameworks to facilitate in creatin light-weight, data-driven approaches which can be trained end-to-end for a given imaging task while only requiring a small amount of training data.

“Correlation-informed ordered dictionary learning for imaging in scattering media” by Chrysoula Tsogka et al. — This Talk also focused on using traditional numerical analysis techniques with a data-driven component, in this case this was Sparce Dictionary Learning using an encoder-decoder network, to compute super-resolution images in a scattering problem.

There were many other fascinating talks, but these were the highlights when it came to the intersection of numerical analysis and ML techniques.

4. Final Thoughts

My time at Waves 2025 was fantastic and provided me with an excellent opportunity to interact with researchers focusing on related problems to my own and using similar ideas for architecture development, as well as a better understanding of the state of the Operator Learning and Scientific Machine Learning within specialist domains like Numerical Analysis.

My biggest take away from Waves 2025 would be that we as Machine Learning (ML) researchers should engage and interact more with domain experts to solve the complex and technical problems which traditional fields, such as modelling and biomedical imaging, face, as opposed to solving these problems on our own using increasingly larger and data-hungry models.

Or in other words, we need more Machine Learning and less Artificial Intelligence as researchers already have many existing tools which work frighteningly well, we should be focusing on improving these methods as opposed to replacing them wholesale with an end-to-end AI model — I found out that an area which many researchers were interested in was uncertainty quantification for outputs of ML and AI models.

Many of the problems which non-ML researchers and specialists are trying to solve have low amounts of training data (and in medical imaging this data is noisy as well), and thus, need solutions which are robust and can be integrated into existing frameworks without requiring an external server of GPUs to function.

Due to this, I encountered a sizable use of self-supervised or semi-supervised learning within the field in order to alleviate the lack of training data and typically a focus on either older techniques — such as dictionary learning and non-variational auto-encoders.

I think that a lot of productive and fruitful work is possible if we as ML researchers could provide domain expertise regarding techniques within ML or AI which supports the work of existing researchers and collaborate on bringing cutting edge advances in the ML and AI fields into existing fields such as numerical analysis and inverse problems.

Overall I had a great time at KIT and in Karlsruhe, it was a wonderful change of pace compared to London and a beautiful and historic city, and the conference provided me a lot of discussion time with fellow researchers and food for thought about possible research directions in the future.

Last month, CDT students travelled to Vancouver to present their work at NeurIPS, one of the largest AI conferences. The schedule was packed, with 6 conference papers and 3 workshop papers presented. The CDT is proud to sponsor and support our PhD students as they present their research at the conference, showcasing their hard work and academic excellence on an international stage!
Day 1, Tuesday: No papers were presented, but the expo, tutorials, and careers fair kept everyone occupied.

Day 2, Wednesday:

William Bankes William and his supervisor presented their work on robust data down sampling. Naive approaches to training on a subset of data can cause problems when classes in the dataset are imbalanced, with rare classes becoming even rarer. A direct result of this is the model’s ability to predict on this subset of the data decreases. To address this they proposed an algorithm called REDUCR, which downsamples data in a manner that preserves the performance of minority classes within the dataset. They show REDUCR works across a range of text and image problems achieving state of the art results. The work is available here.

Jake Cunningham also presented his work on 
Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling

Day 3, Thursday
Yuchen Zhu 

Yuchen’s joint paper with Jialin Yu and Ricardo Silva from UCL Statistical Sciences, Structured Learning of CompositionalSequential Models was presented as part of the main proceedings of NeurIPS 2024. They proposed an explicit model for expressing how the effect of sequential interventions can be isolated into modules, clarifying previously unclear data conditions that allow for the identification of their combined effect at different units and time steps. The paper is here.  Additionally, together with collaborators from MPI Tuebingen, Yuchen presented a paper Unsupervised Causal Abstraction at the Causal Representation Learning workshop. Due to the lack of interpretability of current large blackbox models, they propose a methodology for causally abstracting a large model to a smaller and more interpretable model. In particular, unlike existing methods, their method does not require supervision signals from the smaller model. The paper can be found  here.

David Chanin

David presented a paper along with co-author Daniel Tan, another UCL PhD Student. They find that Contrastive Activation Addition (CAA) steering has mixed results in terms of robustness and reliability. Steering vectors tend to generalise out of distribution when they work in distribution. However, steerability is highly variable across different inputs: depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. While CAA is effective on some tasks, other behaviours turn out to be unsteerable. As a result, it is difficult to ensure they will be effective on a given task of interest, limiting their reliability as a general alignment intervention. The paper is available here.

Day 4, Friday
Reuben Adams

Reuben presented his paper (co-authored with his supervisors) on extending a classic theorem in the PAC-Bayes literature to account for arbitrary outcomes, rather than simply correct or incorrect classifications. Their work provides theoretical insight into the generalisation behaviour of neural networks and the different kinds of errors they can make. Their framework can cover not just Type I and Type II errors, but any kind of error that may occur in multi class classification. You can find the paper here.

Daniel Augusto
Daniel presented a co-authored paper in collaboration with
Getúlio Vargas Foundation for the main conference track. Their work proposes a new solution to streaming variational Bayesian inference using  GFlowNets as a foundation for their methodology. This was the first work that allows high quality variational inference for discrete parameters without requiring the storage of the whole dataset. They believe this work will be useful for applications in genetics, through the inference of phylogenetic trees, for preference learning, and other big-data contexts. Their paper can be read here.

Day 5, Saturday 

Oscar Key Oscar, along with co-authors from Graphcore presented a poster at the workshop on Efficient Natural Language and Speech Processing. Their work considers the top-k operation, which finds the largest k items in a list, and investigates how it can be computed as quickly as possible on the parallel hardware commonly used to run AI applications. Top-k is found in many AI algorithms, so it’s useful to make it fast, e.g. a large language model might use it to select the most important parts of a long prompt. Their full paper is available here.

Varsha Ramineni 

Varsha presented her research at the Workshop on Algorithmic Fairness through the Lens of Metrics and Evaluation (AFME 2024). This work addresses the challenge of evaluating classifier fairness when complete datasets, including protected attributes, are inaccessible. They propose a novel approach that leverages separate overlapping datasets, such as internal datasets lacking demographic information  and external sources like census data, to construct synthetic test data with all necessary variables. Experiments demonstrate that our approach produces synthetic data with high fidelity and offers reliable fairness evaluation where real data is limited.  Varsha says that she had an incredible experience attending her first NeurIPS and presenting her work and engaging in meaningful discussions throughout the conference was a deeply rewarding experience, providing invaluable feedback and ideas as the work extends. Do reach out to her if you’d like to learn more!

On June 12th to 21st of 2024, I had the pleasure to attend and present my work as a poster for the 2nd Bayes-Duality Workshop 2024 organized by the Bayes Duality, a Japan-French joint research project team. This workshop was hosted in the Centre for Advanced Intelligence Project (AIP) of RIKEN in Nihonbashi, Chūō City, Tokyo.

Nihonbashi is one of the oldest districts of Tokyo, a lively business district where finance and general office workers gather while neighbouring the imperial palace, where the Japanese monarch and his family lives. Feeling out of place in this somewhat non-academic environment, the two-week workshop contained invited talks, panels between speakers, showcase of works done by the Bayes Duality team, and a poster session.

As stated in the program, the workshop focused on the development of AI that learns adaptively, robustly, and continuously, like humans. A common theme in the presentations by collaborators of the Bayes Duality is to explore the mathematical connections between the training data examples of and the model parameters of these machine learning systems. This connection is incredibly desirable due to the following difference in complexity: the current state-of-art models have a vast number of uninterpretable parameters while the data examples can usually still be understood by human experts.

Due to the length of the workshop, the invited talks could cover an extensive range of topics. Such breadth of topics is hard to describe in such post and, most incredibly, none of them felt out of place in this workshop. Starting from the expected topics as the tutorial on the Bayesian learning rule, one of the papers that put together the connections between data-parameter duality, and convex duality, to more general topics in uncertainty quantification, such as Eugene Ndiaye’s tutorial and presentation on conformal prediction, continual learning, and identifiability of parameters in neural network models.

The poster session included works mentioned in the invited talks and others from students like me. I chose to present my progress on “Interpretable deep Gaussian processes for geospatial tasks”; in this project I analyse the issue of interpretability of three commonly used architectures of deep Gaussian processes and try to understand what practitioners really meant by “interpretable” and suggest a different metric than the commonly used. I felt this was the right work to present to the audience of this workshop due to their familiarity with Bayesian deep learning and interest in understanding the parameters of these models. As the only student from UCL, I was happy to display our work and connect with researchers from institutions all over the world, with attendees from the US, Asia, and Europe.

The AI & Art competition we ran as part of the CDT Showcase event brought together a fantastic array of talent and creativity, with participants impressing us with their outstanding submissions. We were thrilled to see some innovative approaches and unique perspectives reflected in each entry and are excited to highlight the winning entries that stood out among the rest.

1^st Place: Romy Williamson-The convergence of perception
This piece shows a series of stone busts arranged in a figure. The busts blend smoothly between a perfect sphere, Max Planck, and Igea – the Greek Goddess of Health.
In order to blend smoothly between the busts, I converted the meshes into Spherical Neural Surfaces (read my paper or listen to my talk to find out more) and I optimised a smooth neural map between the two domain spheres, minimising the conformal distortion energy using a variant of the First Fundamental Form.
Romy Comment: the convergence of perception (as named by ChatGPT)
I used our novel shape representation (Spherical Neural Surfaces) to represent the heads of Max Planck and the goddess Igea (converted from meshes), and performed a geometric optimization to find a nice correspondence (diffeomorphism), which then allowed me to interpolate to get the in-between heads.
 This is my paper about Spherical Neural Surfaces: https://arxiv.org/abs/2407.07755 . The geometric optimization part is similar to Neural Surface Maps (https://geometry.cs.ucl.ac.uk/projects/2021/neuralmaps/).

2^nd Place:Reuben Adams – Nook 
The colours in this photo have been subtly changed to encode an audio file of a crackling fireplace, which in turn has been imperceptibly altered to encode a text file of Hardy’s poem The Darkling Thrush. The work telescopes into one image a dreary and wet walk through the peak district, warming by the fire, and thoughts of an old friend.

3^rd Place: Pedro José Ferreira Moreira-UCL Summer School
Welcome to ‘UCL Summer School,’ an exciting comic book adventure that follows a young student on their thrilling journey at University College London Summer School!

Imagine being able to create a whole comic book without knowing how to draw – thanks to AI, that’s exactly what happened here! From packing bags and boarding a plane to sightseeing around London and attending cool AI seminars, this comic capturers every moment with vibrant, dynamic art.
What AI Can Do: AI makes it possible to turn your wildest ideas into reality, even if you can’t draw a stick figure. It helps craft detailed and expressive comic panels that perfectly match the story in your head. Plus, AI is like a super-fast sidekick, helping to create everything in no time!
The Not-So-Great Parts: Sometimes, AI might miss the mark on capturing those deep, personal emotions or might not get the scene just right without some help. It’s great, but it’s not a mind reader – yet!
The Future Is Bright: Imagine a world where AI tools are even more creative, intuitive, and just plain fun to use. We’re talking about easier ways to blend human creativity with AI’s power, making art that’s truly one-of-a-kind.

In ‘UCL Summer School’ you’ll see how AI can turn anyone into a comic book creator, expressing thoughts and stories in a vibrant way that’s never been easier. This comic is all about having fu, exploring new tech and realizing that with a little help from AI, the sky’s the limit for your creativity!
Pedro’s Comments. The motivation behind this comic book art is simple: to show that creativity shouldn’t be limited by technical skills. With the help of AI, anyone can turn their ideas into reality, no matter their experience. Even if you’re “not good at drawing,” you can bring your imagination to life. Sure, the technology isn’t perfect (extra fingers popping up in the art can be a funny surprise), but it’s more than enough to convey emotion and tell captivating stories

4^th Place: Kai Biegun-In With The New
This piece aims to convey a juxtaposition of retro analogue photography and state of the art AI image generation. Four film photos were taken on various film stocks with vintage analogue cameras, and descriptions of those images were used to generate four corresponding photos with the Adobe Firefly image generation suite. I have always felt that the grainy, textured look of film photographs gives them a certain quality that makes looking at them feel like you’re looking at a snapshot from a memory. This is in stark contrast to the saturated, ultra-smooth, somewhat cartoonish look of AI generated photos. I believe this speaks to the fact that, although we are moving towards a world where digital and AI generated media are the norm, there is still place for the analogue to provide a window into real moments, memories, and experiences.
Kai’s comments. The piece is a study of the differences between images captured with analogue cameras and images generated by AI, whereby the analogue photographs were recreated by generative AI by prompting it with a text description of each image. It aims to highlight not just the superficial differences in colour, texture, and subject, but also the difference in feeling one gets from knowing how each image was captured, and question whether that in itself contributes to the artistic merit of the images.

5^th Place: Roberta Chissich-Forest Escape.
Materials Used: Blender 4.1, ANT Landscape Addon, Node Wrangler Addon, Cycles Render Engine, Sapling Tree Gen Addon, Poly Haven Textures.

The Interactive Forest Environment is a meticulously crafted 3D scene designed to immerse viewers in a realist natural landscape. This piece leverages advanced procedural techniques and tools within Blender, reflecting the growing intersection of AI and art in the digital age.

Blender’s geometry nodes and procedural generation tools were extensively used to create the ground and vegetation layouts. These nodes enable the creation of complex, natural-looking terrains and distributions with minimal manual intervention. This results in highly detailed and varied environments without the need for manual modelling of each element. The use of procedural shaders and texture blending techniques in Blender mimics AL-assisted methods to combine ground textures from Poly Haven seamlessly, ensuring enhanced detail and natural transitions.

To optimize rendering, the Cycles Render Engine utilizes NVIDIA’s AI-accelerated denoising technology. OptiX reduces noise in rendered images, significantly speeding up the rendering process while maintaining high-quality visuals. This integration of AI technology helps in producing clean, detailed renders with fewer samples, making the workflow more efficient.

This artwork is inspired by the calming and restorative qualities of nature. It aims to transport viewers to a serene forest environment, providing a momentary escape from the hustle and bustle of everyday life, capturing the essence of nature’s tranquility.
Roberta’s comments. This animated river scene, created in Blender, showcases the power of combining human creativity with advanced tools. By using OptiX rendering, the video achieves a higher level of visual fidelity, capturing the intricate details of light and water. The use of procedural scattering has simplified the placement of grass, leaves, and trees, making the natural landscape come to life effortlessly.
My motivation for this piece comes from the belief that art and technology are not in opposition, but are powerful allies; AI-enhanced tools can aid artists in their creative process. This artwork embodies the idea that we can use these innovations to elevate our creative expression. It’s not about replacing human artistry, it’s about how these tools can help us amplify our imagination, making the impossible possible, and turning complex visions into reality. Together, we can craft a future where human spirit and technological prowess unite to create beauty.

Thank you to everyone who participated. Each entry brought something special to the event and helped create a vibrant and memorable experience for all involved!

In the exciting realm of Model-based Reinforcement Learning (MBRL), researchers are constantly pushing the boundaries of what robots can learn to achieve when given access to an internal model of the environment. One key challenge in this field is ensuring that robots can perform tasks safely and reliably, especially in situations where they lack prior data or knowledge about the environment. That’s where the work of Sicelukwanda Zwane comes into play.

Background

In MBRL, robots use small sets of data to learn a dynamics model. This model is like a crystal ball that predicts how the system will respond to a given sequence of different actions. With MBRL, we can train policies from simulated trajectories sampled from the dynamics model instead of first generating them by executing each action on the actual system, a process that can take extremely long periods of time on a physical robot and possibly cause wear and tear.

One of the tools often used in MBRL is the Gaussian process (GP) dynamics model. GPs are fully-Bayesian models that not only model the system but also account for the uncertainty in state observations. Additionally, they are flexible and are able to learn without making strong assumptions about the underlying system dynamics [1].

The Challenge of Learning Safely

When we train robots to perform tasks, it’s not enough to just predict what will happen; we need to do it safely. As with most model classes in MBRL, GPs don’t naturally incorporate safety constraints. This means that they may produce unsafe or unfeasible trajectories. This is particularly true during early stages of learning, when the model hasn’t seen much data, it can produce unsafe and seemingly random trajectories.

For a 7 degree of freedom (DOF) manipulator robot, bad trajectories may contain self-collisions.

Distributional Trajectory Sampling

In standard GP dynamics models, the posterior is represented in distributional form – using its parameters, the mean vector and covariance matrix. In this form, it is difficult to reason about

about the safety of entire trajectories. This is because trajectories are generated through iterative random sampling. Furthermore, this kind of trajectory sampling is limited to cases where the intermediate state marginal distributions are Gaussian distributed.

Pathwise Trajectory Sampling

Zwane uses an innovative alternative called “pathwise sampling” [3]. This approach draws samples from GP posteriors using an efficient method called Matheron’s rule. The result is a set of smooth, deterministic trajectories that aren’t confined to Gaussian distributions and are temporally correlated.

Adding Safety

The beauty of pathwise sampling [3] is that it has a particle representation of the GP posterior, where individual trajectories are smooth, differentiable, and deterministic functions. This allows for the isolation of constraint-violating trajectories from safe ones. For safety, rejection sampling is performed on trajectories that violate safety constraints, leaving behind only the safe ones to train the policy. Additionally, soft constraint penalty terms are added to the reward function.

Sim-Real Robot Experiments

To put this approach to the test, Zwane conducted experiments involving a 7-DoF robot arm in a simulated constrained reaching task, where the robot has to avoid colliding with a low ceiling. The method successfully learned a reaching policy that adhered to safety constraints, even when starting from random initial states.

In this constrained manipulation task, the robot is able to reach the goal (shown by the red sphere – bottom row) without colliding with the ceiling (blue – bottom row) using less than 100 seconds of data in simulation.

Summary

Sicelukwanda Zwane’s research makes incremental advances on the safety of simulated trajectories by incorporating safety constraints while keeping the benefits of using fully-Bayesian dynamics models such as GPs. This method promises to take MBRL out of simulated environments and make it more applicable to real-world settings. If you’re interested in this work, we invite you to dive into the full paper, published at the recent IEEE CASE 2023 conference.

References

1. M. P. Deisenroth and C. E. Rasmussen. PILCO: A Model-based and Data-efficient Approach to Policy Search. ICML, 2011.
2. S. Kamthe and M. P. Deisenroth. Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control. AISTATS, 2018.
3. J. T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Pathwise Conditioning of Gaussian Processes. JMLR, 2021.