Schedule for: 25w5469 - Mathematical Analysis of Adversarial Machine Learning

Creating Artificial Super Intelligence (ASI) (AI that surpasses human intelligence) is the ultimate challenge in AI research. This is, as we show, fundamentally linked to the problem of avoiding hallucinations (wrong, yet plausible answers) in AI. We discover a key mechanism that must be present in any ASI. This mechanism is not present in any modern chatbot and we establish that without it, ASI will never be achievable. Moreover, we reveal that AI missing this mechanism will always hallucinate. The mechanism we introduce is the computation of what we call an `I don't know' function. An `I don't know' function determines when an AI is correct and when it will not be able to answer with 100% confidence. The root to these findings is the Consistent Reasoning Paradox (CRP) that we present, which is a paradox in logical reasoning. The CRP shows that the above mechanism must be present as -- surprisingly -- an ASI that is `almost certain' (more than 50%) can rewrite itself to become 100% certain. It will compute an `I don't know' function and either be correct with 100% confidence, or it will not be more than 50% sure. The CRP addresses a long-standing issue that stems from Turing's famous statement that infallible AI cannot be intelligent, where he questions how much intelligence may be displayed if an AI makes no pretence at infallibility. The CRP reveals the answer -- consistent reasoning requires fallibility -- and thus marks a necessary fundamental shift in AI design if ASI is to ever be achieved and hallucinations to be stopped.

We present a notion of λ-monotonicity for an n-species system of PDEs governed by flow dynamics, extending monotonicity in Banach spaces to the Wasserstein-2 metric space. We show that monotonicity implies the existence of and convergence to a unique steady state. In the special setting of Wasserstein-2 gradient descent of different energies for each species, we prove convergence to the unique Nash equilibrium of the associated energies and discuss the relationship between monotonicity and displacement convexity. This extends known zero-sum (min-max) results in infinite-dimensional game theory to the general-sum setting. We provide examples of monotone coupled gradient flow systems, including cross-diffusion, nonlocal interaction, and linear and nonlinear diffusion. Numerically, we demonstrate convergence of a four-player economic model for market competition, and an optimal transport problem. This is joint work with Ricardo Baptista, Franca Hoffmann, Eric Mazumdar, and Lillian Ratliff.

In adversarially robust classification problems, one expects that the inclusion of an adversary will narrow the space of possible solutions and will enforce some degree of smoothness upon classifiers. For an important class of non-parametric, adversarially robust classification problems formal arguments strongly suggest that epsilon-robust optimal classifiers should have $C^2$ smoothness on the order of $1/epsilon$, but this has been challenging to prove. This talk will discuss recent work (with Rachel Morris), which proves this type of regularity of adversarially robust classifiers in two dimensions. This work is inspired by properties of classical perimeter minimizers, and utilizes a detailed analysis via second variations in order to rule out non-trivial normal cones of the optimal classifiers. The talk will also discuss some of the technical barriers to extending this result to higher dimensions.

Adversarial training is a framework widely used by practitioners to enforce robustness of machine learning models. During the training process, the learner is pitted against an adversary who has the power to alter the input data. As a result, the learner is forced to build a model that is robust to data perturbations. Despite the importance and relative conceptual simplicity of adversarial training, there are many aspects that are still not well-understood (e.g. regularization effects, geometric/analytic interpretations, tradeoff between accuracy and robustness, etc...), particularly in the case of multiclass classification. In this talk, I will show that in the non-parametric setting, the adversarial training problem is equivalent to a generalized version of the Wasserstein barycenter problem. The connection between these problems allows us to completely characterize the optimal adversarial strategy and to bring in tools from optimal transport to analyze and compute optimal classifiers. This also has implications for the parametric setting, as the value of the generalized barycenter problem gives a universal upper bound on the robustness/accuracy tradeoff inherent to adversarial training.

We establish a characterization of the subdifferential of a nonlocal total variation which is one-homogeneous and arises in a convex relaxation of the adversarial training problem for binary classification. To achieve this, we derive a dual representation of the nonlocal total variation functional involving a nonlocal divergence and gradient and ensure the consistency with their local counterparts. Using the dual representation, we apply a primal-dual algorithm to the relaxed adversarial training problem to provide an efficient algorithmic approach to solve the original problem. Lastly, we perform a numerical analysis of the implementation comparing it with state of the art methods for solving the adversarial training problem.

We explore properties of optimal adversarial strategies and optimal robust classifiers of adversarial training in multiclass classification with an agnostic learner. Building on reformulations as generalized barycenter problem and multimarginal optimal transport problem from previous works, we prove that under stronger conditions of quadratic cost and data distributions with full support in a balanced class setting: 1) The optimal adversarial strategy, which can be viewed as a generalized barycenter, is unique with full support. 2) The optimal robust classifiers constructed from dual solutions of multimarginal optimal transport problem are regular with bounded Hessian. Examples with synthetic data of binary classification illustrate our results.

Randomization has emerged as a promising strategy for improving the adversarial robustness of machine learning models. While much of the existing theory focuses on binary classification, the multiclass setting remains less understood. In this talk, we will discuss recent work that aims to shed light on this area by drawing connections to ideas from graph theory. By focusing on discrete data distributions, we frame adversarial risk minimization in terms of set packing problems, which offers a structured way to analyze when and how randomization might help. Through this lens, we identify certain conditions on the data distribution that appear necessary for randomized strategies to offer improvements. This allows us to construct data distributions for which randomized models outperform their deterministic counterparts in the multiclass classification setting. These insights underscore the often-overlooked power of randomization in defending against adversarial attacks and open interesting lines of research for further theoretical study.

Algorithms for distributionally robust optimization (DRO) are categorized by the probability divergences they use for constructing ambiguity sets, e.g., the Kullback-Leibler divergence and the Kantorovich-Wasserstein metric. KL-divergence based DRO was initially studied and later criticized for its shortcomings compared with the Wasserstein distance. In this talk, I will first contrast the Wasserstein DRO with the KL and Fisher-Rao types of DRO, showcasing their distinct analytical properties. Such analysis is based on recent advances in the mathematical foundation of the Hellinger and Fisher-Rao type gradient flows, mirroring the correspondence between the Wasserstein DRO and gradient flows. To go beyond mere theoretical insights, I will then showcase some new DRO-type algorithms motivated by the analysis of gradient flows in the Fisher-Rao type of geometries.

Supervised Transfer Learning (STL) concerns scenarios where a learner has access to some labeled data for a prediction task, a much labeled data from a different but related distribution. The goal of the leaner is to optimally leverage both datasets towards improved prediction on the target task. We are interested in the extent to which existing notions of distance or discrepancy between distributions might capture the statistical hardness of this problem, and design efficient procedures that can automatically adapt to a priori unknown such distances between distributions.

Many applications, especially in financial mathematics, can be formulated as classification tasks whose inputs are finite-horizon, discrete-time paths, and for which the temporal dependence structure of the data is essential. A canonical example is the automatic detection of buy-versus-sell signals in algorithmic trading. This talk investigates how to train classifiers that remain robust when the true data-generating process lies in an ambiguity set defined by the adapted (nested) Wasserstein distance between probability laws on paths (see e.g. Pflug and Pichler (2012)). This metric extends the standard Wasserstein distance by incorporating the underlying filtration, thereby respecting the information flow in time. We present preliminary results that recast the resulting distributionally-robust classification problem as the evaluation of a certain dynamic risk measure. This reformulation generalises earlier links between static Wasserstein ambiguity and coherent risk measures such as CVaR (for example as in Nguyen and Wright (2021) or Blanchet and Murthy (2017)). Numerical experiments illustrate the relevance of this connection. This is joint work with Ling Chen.

The MBO scheme (originally due to Merriman, Bence, and Osher in a different context) is an efficient method for data clustering and classification. Its basic version alternates between diffusion and pointwise thresholding. In this talk, I will present several recent analytical results in collaboration with Jona Lelmi and Anton Ullrich in the regime of large data clouds and small (time) steps that draw connections to geometric flows.

Performative learning addresses the increasingly pervasive situations in which algorithmic decisions may induce changes in the data distribution as a consequence of their public deployment. We propose a novel view in which these performative effects are modelled as push-forward measures. This general framework encompasses existing models and enables novel performative gradient estimation methods, leading to more efficient and scalable learning strategies. For distribution shifts, unlike previous models which require full specification of the data distribution, we only assume knowledge of the shift operator that represents the performative changes. Focusing on classification with a linear-in-parameters performative effect, we prove the convexity of the performative risk under a new set of assumptions. Notably, we do not limit the strength of performative effects but rather their direction, requiring only that classification becomes harder when deploying more accurate models. In this case, we also establish a connection with adversarially robust classification by reformulating the minimization of the performative risk as a min-max variational problem.

Adversarial attacks pose significant challenges in many machine learning applications, particularly in the setting of distributed training and federated learning, where malicious agents seek to corrupt the training process with the goal of jeopardizing and compromising the performance and reliability of the final models. In this talk, we address the problem of robust federated learning in the presence of such attacks by formulating the training task as a bi-level optimization problem. We conduct a theoretical analysis of consensus-based bi-level optimization (CB2O), an interacting multi-particle metaheuristic optimization method (presented by Konstantin Riedl), and investigate its resilience in adversarial settings (presented by Sixu Li). Specifically, we provide a global convergence analysis of CB2O in mean-field law without and with the presence of malicious agents, demonstrating the robustness of CB2O against a diverse range of attacks. On the practical side, we extend CB2O to the clustered federated learning setting by proposing FedCB2O, a novel interacting multi-particle system, and design a practical algorithm that addresses the demands of real-world applications. Extensive experiments demonstrate the robustness of the FedCB2O algorithm against label-flipping attacks in decentralized clustered federated learning scenarios, showcasing its effectiveness in practical contexts. This talk is based on joint works of José A. Carrillo, Nicolás García Trillos, Aditya Kumar Akash, Sixu Li, Konstantin Riedl, and Yuhua Zhu. J. A. Carrillo, N. García Trillos, S. Li, and Y. Zhu. FedCBO: Reaching group consensus in clustered federated learning through consensus-based optimization. Journal of Machine Learning Research, 25(214):1–51, 2024. N. García Trillos, S. Li, K. Riedl, and Y. Zhu. CB2O: Consensus-based bi-level optimization. arXiv preprint arXiv:2411.13394, 2024. N. García Trillos, A. Kumar Akash, S. Li, K. Riedl, and Y. Zhu: Defending against diverse attacks in federated learning through consensus-based bi-level optimization. Phil. Trans. R. Soc. A.383:20240235, 2025.

Adversarial attacks pose significant challenges in many machine learning applications, particularly in the setting of distributed training and federated learning, where malicious agents seek to corrupt the training process with the goal of jeopardizing and compromising the performance and reliability of the final models. In this talk, we address the problem of robust federated learning in the presence of such attacks by formulating the training task as a bi-level optimization problem. We conduct a theoretical analysis of consensus-based bi-level optimization (CB2O), an interacting multi-particle metaheuristic optimization method (presented by Konstantin Riedl), and investigate its resilience in adversarial settings (presented by Sixu Li). Specifically, we provide a global convergence analysis of CB2O in mean-field law without and with the presence of malicious agents, demonstrating the robustness of CB2O against a diverse range of attacks. On the practical side, we extend CB2O to the clustered federated learning setting by proposing FedCB2O, a novel interacting multi-particle system, and design a practical algorithm that addresses the demands of real-world applications. Extensive experiments demonstrate the robustness of the FedCB2O algorithm against label-flipping attacks in decentralized clustered federated learning scenarios, showcasing its effectiveness in practical contexts. This talk is based on joint works of José A. Carrillo, Nicolás García Trillos, Aditya Kumar Akash, Sixu Li, Konstantin Riedl, and Yuhua Zhu. J. A. Carrillo, N. García Trillos, S. Li, and Y. Zhu. FedCBO: Reaching group consensus in clustered federated learning through consensus-based optimization. Journal of Machine Learning Research, 25(214):1–51, 2024. N. García Trillos, S. Li, K. Riedl, and Y. Zhu. CB2O: Consensus-based bi-level optimization. arXiv preprint arXiv:2411.13394, 2024. N. García Trillos, A. Kumar Akash, S. Li, K. Riedl, and Y. Zhu: Defending against diverse attacks in federated learning through consensus-based bi-level optimization. Phil. Trans. R. Soc. A.383:20240235, 2025.

The Merriman–Bence–Osher (MBO) scheme with volume constraints offers a computationally efficient approximation to volume-preserving mean curvature flow. This method has shown strong empirical performance in data-driven applications such as clustering and semi-supervised classification due to its geometric approach. The talk will briefly discuss computational aspects and their connection to the consistency of the scheme in the regime of vanishing time step and increasing data size.

This talk explores legal rules that apply to adversarial attacks on machine learning through the lens of United States law. Building on research done across a number of years with my colleagues Ram Shankar Siva Kumar and Jon Penney, I fit adversarial attacks on machine learning systems into the broader frame of computer security, and discuss how certain interpretations of existing anti-hacking laws, like the Computer Fraud and Abuse Act, can disincentivize for the development of defenses and robust models.

A major part of the success of deep learning can be attributed to the exceptional performance of advanced classifiers like feed-forward networks in difficult scenarios such as image classification. Despite this, the widespread presence of adversarial attacks has remained problematic, especially in scenarios where trustworthy AI is necessary. There have been many attempts to rectify this issue, however, up until now none have been universally successful. In this talk we attempt to identify the root cause of this issue. We do this by considering theoretical results which ensure that, for any fixed dimensions (number of neurons, number of layers) for a feed-forward neural network, it is possible to create classification tasks that are both inherently stable and for which we can train a neural network that will have exceptional performance both on the training set and in their ability to generalise. However, these trained neural networks have an inherent weakness – they must be unstable. This goes some way to explaining why it has been so challenging to address adversarial attacks and provides an outlook on what must occur to overcome this barrier. In this talk, we will discuss these challenges and how to rectify them. In particular, our results show the potentially surprising result that the size of relevant datasets must be linked to the number of neurons required to create stable neural networks.

We study learning in an adversarial setting, where an epsilon fraction of samples from a distribution P are globally corrupted (arbitrarily modified), and the remaining perturbations have an average magnitude bounded by rho (local corruptions). With access to n such corrupted samples, we aim to develop a computationally efficient approach that achieves the optimal minimax excess risk. Our approach combines a data-driven cleaning module with a distributionally robust optimization (DRO) framework. We demonstrate that if the data cleaning module is minimax optimal with respect to the Wasserstein loss, solving an optimal transport-based DRO problem ensures a minimax optimal decision. We further provide tractable reformulations for both modules. Specifically, we introduce an optimal filtering algorithm to clean corrupted data by identifying and removing outliers. For the DRO module, we reformulate the problem as a two-player zero-sum game, deriving finite convex formulations. We show that the minimax theorem applies to this game, and Nash equilibria exist. Finally, we present a principled approach for constructing adversarial examples.

Artificial intelligence is currently leading to one breakthrough after the other, in industry, public life, and the sciences. Yet significant challenges remain, especially for industrial and safety-critical applications. Two of the most pressing issues are the lack of reliability as well as trustworthiness of AI methods and the massive energy consumption of current AI technologies. In this lecture, we will provide an introduction to these challenges from a mathematical perspective. We will present some of our recent advances in the development of reliable and trustworthy AI, with a particular focus on generalization and explainability. We will then turn to the topic of sustainable AI, addressing the issue of energy efficiency. This discussion naturally leads us to the field of analog AI systems, such as neuromorphic computing and the corresponding models of spiking neural networks. We will conclude by highlighting their potential for building AI systems that are both energy-efficient and trustworthy.

AI agents operating in the real world must grapple with a persistent lack of data in dynamic, ever-changing environments. Effective interactive decision-making requires moving beyond knowledge distillation: an intelligent agent must not only act but also recognize and resolve its own uncertainty. Despite their impressive capabilities in basic knowledge work, state-of-the-art AI systems continue to struggle with articulating uncertainty—for example, OpenAI recently acknowledged that its latest agentic system, DeepResearch, “often fails to convey uncertainty accurately.” This talk presents a series of recent works addressing the core challenge of uncertainty quantification in natural language-based interactive decision-making. Rather than modeling latent environment parameters, we conceptualize uncertainty as stemming from unobserved future outcomes and quantify it through autoregressive sequence generation: iteratively predicting the next outcome conditioned on the past. By adapting to new information via in-context learning instead of relying on cumbersome posterior inference, our approach naturally scales to problems involving unstructured data, such as adaptive student assessment incorporating text and images. Formally, we establish a reduction from online decision-making to offline next-outcome prediction, enabling us to leverage the vast datasets and computational infrastructure developed for sequence modeling to advance interactive decision-making capabilities.

Prior studies recast the adversarial learning problem as a primal-dual game, thereby suggesting the applicability of primal-dual optimization algorithms. We show that full-batch adversarial training on a family of convex problems with ridge regularization is formally equivalent to a primal-dual Frank-Wolfe algorithm. This perspective yields improved step-size selection and a principled approach to mini-batching.

In safety-critical applications, machine learning models must generalize to new target distributions reliably under worst-case distribution shifts. Recent methods leveraging structural invariances in the data-generating process have shown that infinite robustness, i.e., robustness to shifts of arbitrary magnitude, is achievable, but only when the invariant structure is fully identifiable. However, such guarantees require strong assumptions on the number and diversity of available training data distributions, which are often unmet in practice. This mismatch frequently leads to the failure of invariance-based methods on real-world data. In this talk, we explore the extent of distributional robustness that remains achievable under partial identifiability of invariant representations, focusing on simple regression and classification tasks. Finally, we discuss how access to few target samples can enable domain adaptation under partial identifiability when source-only guarantees are too pessimistic.

As machine learning systems scale in both model complexity and data volume, they increasingly rely on distributed algorithms to process massive datasets efficiently. Yet, with this scale comes vulnerability: data from diverse sources may be noisy, corrupted, or adversarial, and distributed computing environments are prone to hardware faults, software bugs, and malicious attacks. If left unaddressed, these issues can significantly undermine the reliability of large-scale distributed learning systems. In this talk, I will discuss how to design robust learning algorithms that remain reliable in the face of such real-world challenges. Focusing on stochastic gradient descent (SGD), the foundation of modern ML optimization, I will present recent advances in robust aggregation methods and examine how robustness interacts with data heterogeneity in distributed settings. I will highlight key distinctions between robust machine learning and classical robust statistics, and conclude with some open problems and research directions at the intersection of theory and practice.

We discuss some qualitative and quantitative results analyzing adversarial training as the adversarial budget $\epsilon$ vanishes. First, with Bungert, we find that minimizers of the adversarial training problem converge in $L^1$ to a Bayes classifier that has minimal weighted perimeter, showing that adversarial training acts as a selection mechanism for the standard classification problem. Subsequent work by Morris and Murray showed that adversarial minimizers converge to Bayes classifiers in the (much stronger) Hausdorff metric at a rate that degrades with the dimension. Joining forces, we recover the generically optimal rate of convergence showing that for all $\epsilon$, the Hausdorff distance between the adversarial minimizer and the Bayes classifier is $O(\epsilon)$ regardless of the ambient dimension. Based on joint work with L. Bungert, R. Morris, and R. Murray.