By Laura Ruis, PhD Candidate

Recently, Randall Balestriero, Jerome Pesenti, and Yann LeCun dropped a paper on arXiv that clarifies certain terms that are often used when people talk about generalization in machine learning. In machine learning, we often formulate a differentiable objective function for our problem that we can optimize with gradient-based methods. We tune model parameters given data such that this objective function is optimized. However, what differentiates machine learning from optimization is that we do not just want our model to optimize the objective function for the data we used to learn the parameters, called the training data, we also want it to generalize to unseen data points. Modern machine learning methods have become very good at this for lots of applications like speech recognition, machine translation, and image classification. However, some people claim (here, here, and here) that these methods are simply interpolating the training data they see during training, and that they would fail when classifying a new data point requires extrapolation. The paper by Balestriero et al. shows that this is not the case for a specific definition of interpolation and extrapolation. They come to the following conclusion:

We shouldn’t use interpolation/extrapolation in the way the terms are defined in the paper when talking about generalization, because for high dimensional data deep learning models always have to extrapolate, regardless of the dimension of the underlying data manifold. 

In this post I’ll attempt to shed some light on this conclusion. It’s drawn in part from the first figure in the paper, which we will reproduce from scratch. In the process of doing that we’ll encounter all the relevant background material that’s necessary to understand this paper. I’ll go through all the code and maths that’s required to reproduce it. Below you can see the figure I’m talking about, that without any explanation won’t illuminate much yet. If you want to understand it, read on!

At the end this post, we will know more about the following terms:

• The curse of dimensionality
• Convex hull
• Ambient dimension
• Intrinsic dimension / data manifold dimension
• Interpolation / extrapolation

Click here to read full post.

The UKRI Centre for Doctoral Training in Foundational AI were delighted to welcome our third cohort of students at the start of term. We are now training 33 PhD students in foundational artificial intelligence, helping to create new algorithms and AI coding to help answer some of society’s biggest needs.  

We have slowly started to return to the Centre for Artificial Intelligence at 90 High Holborn and the purchase of a table tennis table and a table football table has certainly provided some welcome light relief between serious study. The idea of league tables and leader boards has been mooted, so watch this space! 

We are incredibly proud of our space in Holborn and the CDT itself and have been working with UCL media services to create a promotional video, highlighting the CDT and AI Centre. We were incredibly fortunate to have students and staff give up their time to share their experiences and journey at UCL and are really excited to see the finished product.  

One of the aims of the video is to encourage new applicants from existing MSc students around the United Kingdom. The UKRI Centre for Doctoral Training in Foundational AI is funded by the EPSRC and aims to create the next generation of UK AI creators and innovators.  The UK government has released an ambitious National AI Strategy of building upon our rich cultural history in machine learning to lead the field of next generation AI and AI driven industry.  Applications are now being accepted for the 2022 entry, so please do share this information with any interested parties.  

What makes the CDT stand out from other PhD providers is the cohort activities and events that are planned throughout the year for our students.  These include weekly seminars with guest speakers, workshops, training, internship opportunities and equality, diversity and inclusion events.  This calendar year we have put on event to celebrate the LGBTQ+ community, Women in AI, supporting the black community in AI and we end with #WeThe15 in AI, an event to raise the voices of those within the disabled community.  

Our event will take place virtually on 2nd December from 1pm-5pm and tickets are available now via Eventbrite: https://www.eventbrite.co.uk/e/207803364457  

We are delighted to provide a platform for those who have lived experiences as part of the disabled community, as well as those doing ground-breaking research to provide technological solutions to human problems.

Our keynote speaker is Professor Catherine Holloway (UCL), Academic Director, Global Disability Innovation Hub.

Schedule of events 

13:00 Welcome 

13:05 Presentation by Luis Canto E Castro from City Maas – a company that uses AI to help wheelchair users navigate city landscapes 

13:30 Presentation William Dudley, Imperial College London – An AI research student at Imperial College London who has cerebral palsy and will share his lived experience as a member of the disabled community and researching in AI 

13:55 Presentation Prof Marie Schaer & Dr Thomas Maillart, University of Geneva – Looking at their recent paper using AI image processing to help diagnose autism in young children.  

14:20 Panel -Autism and AI 

Chair Alice Renard, Autistic Society, UCL 

Panellists: Suzanna Chen (Autistic Society, UCL), Luke Muschialli (Autistic Society, UCL), Dr Larissa Romualdo Suzuki (Head of Data and AI, Google Cloud), Dr Joe Mintz (Institute of Education, UCL), Prof Marie Schaer (University of Geneva) and Dr Thomas Maillart (University of Geneva) 

15:15 Key Note: Prof Catherine Holloway, UCL – “Disability Innovation: AI and Assistive Technologies”  

15:40 Panel: AI-Powered Disability Innovation  

Chair: Daniel Hajas, GDI Hub Innovation Manager at GDI Hub 

Panellists: Mr Klaus Höckner, CEO, Austrian Association for Blind and Visually Impaired, Mr Bernard Chiira, Director at Innovate Now, Co-Founder Inclusive Education Network, Prof Aldo Faisal, Imperial College London 

More panellists TBC 

16:35 Presentation by Prof Aldo Faisal, Imperial College London – “AI for Disabilities: from Assistive Technology to Human Augmentation” 

17:05 Closing remarks 

 Please do register for this event and let us know what other events you would like us to share with you.

Welcome to the blog for the UKRI Centre for Doctoral Training in Foundational Artificial Intelligence.

Our aim with this blog is to inform the wider world of our research, ourselves and our ambitions in helping create new algorithms and foundational practices in artificial intelligence to help deliver the UK National Strategy in artificial intelligence.

Our CDT is one of 16 UKRI funded CDT’s focusing on artificial intelligence and building on the UK’s history of excellence with machine learning.

The UKRI CDT in Foundational AI sits within UCL’s Centre for Artificial Intelligence in the heart of London and helps bring together some of the best minds in the field of machine learning, natural language processing, robotics, deep learning and reinforcement learning (and so much more!)

We look forward to letting you know more about us and what we are doing to help forward the research in artificial intelligence and create new frontiers in research.