X Close

UCL Department of Science, Technology, Engineering and Public Policy


Applied in Focus. Global in Reach


Archive for the 'Public Policy Processes and Knowledge Systems' Category

Adversarial Attacks, Robustness and Generalization in Deep Reinforcement Learning

By Ezgi Korkmaz, on 20 December 2023

Reinforcement learning has achieved substantial progress on successfully completing tasks, from solving complex games to large language models (i.e. GPT-4) including many different fields from medical applications to self-driving vehicles and finance, by learning from raw high-dimensional data with the utilization of deep neural networks as function approximators.

The vulnerabilities of deep reinforcement learning policies against adversarial attacks have been demonstrated in prior studies [1,2,3,4]. However, a recent study takes these vulnerabilities one step further and introduces natural attacks (i.e. natural changes to the environment given that these changes are imperceptible) while providing a contradistinction between adversarial attacks and natural attacks. The instances of such changes include, but are not limited to creating a blur, introduction of compression artifacts, or perspective projection of the state observations at a level that humans cannot perceive the change.

Intriguingly, the results reported demonstrate that these natural attacks are at least equally, and often more imperceptible compared to adversarial attacks, while causing larger drop in policy performance. While these results carry significant concerns regarding artificial intelligence safety [5,6,7], they further raise questions on the model’s security. Note that the prior studies on adversarial attacks on deep reinforcement learning rely on the strong adversary assumption, in which the adversary has access to the policy’s perception system, training details of the policy (e.g. algorithm, neural network architecture, training dataset), and the ability to alter observations in real time with simultaneous modifications to the observation system of the policy with computationally demanding adversarial formulations. Thus, the fact that natural attacks described in [8] are black-box adversarial attacks, i.e. the adversary does not have access to the training details of the policy and the policy’s perception system to compute the adversarial perturbations, raises further questions on machine learning safety and responsible artificial intelligence.

Furthermore, the second part of the paper investigates the robustness of adversarially trained deep reinforcement learning policies (i.e. robust reinforcement learning) under natural attacks, and demonstrates that vanilla trained deep reinforcement learning policies are more robust than adversarially trained policies. While these results reveal further security concerns regarding the robust reinforcement learning algorithms, they further demonstrate that adversarially trained deep reinforcement learning policies cannot generalize at the same level as straightforward vanilla trained deep reinforcement learning algorithms.

This study overall, while providing a contradistinction between adversarial attacks and natural black-box attacks, further reveals the connection between generalization in reinforcement learning and the adversarial perspective.

Author’s Note: This blog post is based on the paper ‘Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness’ published in AAAI 2023.
[1] Adversarial Attacks on Neural Network Policies, ICLR 2017.
[2] Investigating Vulnerabilities of Deep Neural Policies. Conference on Uncertainty in Artificial Intelligence (UAI).
[3] Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs. AAAI Conference on Artificial Intelligence, 2022. [Paper Link]
[4] Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisions. International Conference on Machine Learning, ICML 2023. [Paper Link]
[5] New York Times. Global Leaders Warn A.I. Could Cause ‘Catastrophic’ Harm, November 2023.
[6] The Washington Post. 17 fatalities, 736 crashes: The shocking toll of Tesla’s Autopilot, June 2023.
[7] The Guardian. UK, US, EU and China sign declaration of AI’s ‘catastrophic’ danger, November 2023.
[8] Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness, AAAI Conference on Artificial Intelligence, 2023. [Paper Link]
[9] Understanding and Diagnosing Deep Reinforcement Learning. International Conference on Machine Learning, ICML 2024.

The best intergovernmental platform that you’ve never heard of… until now

By c.washbourne, on 7 May 2019

Last week saw hundreds of people gather in Paris for what some have described as the ‘IPCC (Intergovernmental Panel on Climate Change) for nature’.

The ‘Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services’, or IPBES to its friends, works to produce knowledge on the state of nature and to support the development of skills and capacity to promote its sustainable use. As an independent, intergovernmental body, IPBES spends much of its time building the policy case for biodiversity and ecosystem services; helping policy-makers to make more informed decisions for a sustainable future.

So why, might you ask, have you never heard anything about IPBES? Good question!

I was privileged to attended the most recent plenary meeting of the platform ‘IPBES-7’, which took place last week in Paris, as part of the delegation of the Young Ecosystem Services Specialists (YESS), and can offer a few possible answers.


Perspectives on the STEaPP Internship: Public policy processes and knowledge systems

By ucqnafe, on 17 July 2017

alessandro-allegra-smallSTEaPP Intern Alessandro Allegra takes stock two-thirds of the way into his internship working on research project Public policy processes and knowledge systems.

I have been working on a project with Dr Chris Tyler and Dr Adam Cooper to investigate public policy processes and knowledge systems. The first thing I had to do for the project was figure out what exactly we mean by a ‘policy process’. This was not just because of my ignorance, but because it turns out that, although there are no shortage of definitions and diagrams out there describing the policy process, it turns out that they often have very little resemblance with happens in the corridors of Whitehall.

By surveying the existing literature, analysing the paper trail behind some specific policy initiatives, and talking to people who have been involved in government policy in various roles, I have started building a more nuanced picture. Rather than a neat and orderly cycle of well-distinguished stages, the process seems to be more composed of phases that blend into each other, often with iterations and feedback loops, where the boundaries can only be drawn retrospectively through post-hoc rationalisation.

This more nuanced understanding of what happens at the coalface of policymaking is so far the greatest lesson that I have learned during my time at STEaPP, and has several implications for how we understand the role of scientific knowledge in it. The next step of the project will be bringing these insights together into a coherent model, and validate it though discussion with practitioners. This will then allow to ask questions about the use for evidence in policy from a different perspective, such as for example what pressures and constraints civil servants encounter in their daily policy work, what activities and cognitive processes they engage with, and how vulnerable the whole process is to individual cognitive biases we are inevitably victim of.

STEaPP Research Internships 2018

To find out about details of our internship programme for 2018, visit the internship webpages.