Schedule for: 25w5414 - Bridging Theory and Practice in Finance

A set dinner is served daily between 5:30pm and 7:30pm in the Xianghu Lake National Tourist Resort.

When dealers hedge large currency fix exposures on behalf of their clients, this can lead to predictable price patterns that opportunistic traders can exploit. We show that the cost of this information leakage is predominantly borne by the client, but that it can be partially mitigated when the dealer hedges some of their exposure ahead of the fixing window. The dealer also benefits from such a strategy because it lets them average in at a hedging price that is favourable compared to where the fix settles. Information leakage therefore mitigates a conflict of interest that would otherwise exist by aligning the interests of the dealer and the client against those of the opportunistic trader. (Joint work with Roel Oomen (Deutsche Bank) and Mateo Rodriguez Polo (ETH Zurich))

Passive liquidity providers (LPs) in automated market makers (AMMs) face losses due to adverse selection, which static trading fees often fail to offset in practice. We study the key determinants of LP profitability in a dynamic reduced-form model where an AMM operates in parallel with a centralized exchange (CEX), traders route their orders optimally to the venue offering the better price, and arbitrageurs exploit price discrepancies. Using large-scale simulations and real market data, we analyze how LP profits vary with market conditions such as volatility and trading volume, and characterize the optimal AMM fee as a function of these conditions. The mechanisms driving these relationships are highlighted through extensive comparative statics, and the model’s relevance is confirmed through calibration to market data. We find that under normal market conditions, the optimal AMM fee is competitive with the trading cost on the CEX and remarkably stable, whereas in periods of very high volatility, a high fee protects passive LPs from severe losses. These findings suggest a threshold-type dynamic fee schedule to improve LP outcomes. (Joint work with Steven Campbell, Philippe Bergault, Jason Milionis)

We study optimal stopping for diffusion processes with unknown model primitives within the continuous-time reinforcement learning (RL) framework developed by Wang et al (2020), and present applications to option pricing and portfolio choice. By penalizing the corresponding variational inequality formulation, we transform the stopping problem into a stochastic optimal control problem with two actions. We then randomize controls into Bernoulli distributions and add an entropy regularizer to encourage exploration. We derive a semi-analytical optimal Bernoulli distribution, based on which we devise RL algorithms using the martingale approach established in Jia and Zhou (2022). We prove a policy improvement theorem and a fast convergence of the resulting policy iterations. We demonstrate the effectiveness of the algorithms in pricing finite-horizon American put options, solving Merton's problem with transaction costs, and scaling to high-dimensional optimal stopping problems, by showing that both the offline and online algorithms achieve high accuracy in learning the value functions and characterizing the associated free boundaries. Joint work with Min Dai, Yu Sun and Zuoquan Xu. 

Optimal transport theory provides a powerful analytical framework for studying probability measures. However, it falls short when applied to the laws of stochastic processes, as it ignores their intrinsic temporal structure — namely, the filtration. Recently, an adapted optimal transport framework has emerged as a natural extension of classical optimal transport to the setting of stochastic processes. In this talk, I will focus on barycenters in this context and highlight their relevance to consensus models in mathematical finance.

Lunch is served daily between 11:30am and 1:30pm in the Xianghu Lake National Tourist Resort

We study model-based reinforcement learning for optimal control problems in diffusion environments. An important question is how to learn a good model for decision making, for which standard statistical methods may not perform well. We propose a novel value-aware framework for model learning, where we minimize the mismatch between the model-based value function and empirical rewards. We develop theoretical results such as characterization of identifiability and convergence and asymptotic distribution of the estimator. To perform value-aware model estimation in general problems, we construct a surrogate loss that can be efficiently optimized even in high dimensions and provide theoretical guarantee on the convergence of its minimizer. We show that our approach can reduce decision bias caused by model misspecification and identify weak signals from noisy environments in some financial applications.

This study develops a BSDE method for the long-term decomposition of pricing kernels under the G-expectation framework. We establish the existence, uniqueness, and regularity of solutions to three types of quadratic G-BSDEs: finite-horizon G-BSDEs, infinite-horizon G-BSDEs, and ergodic G-BSDEs. Moreover, we explore the Feynman--Kac formula associated with these three types of quadratic G-BSDEs. Using these results, a pricing kernel is uniquely decomposed into four components: an exponential discounting component, a transitory component, a symmetric G-martingale, and a decreasing component that captures the volatility uncertainty of the G-Brownian motion. Furthermore, these components are represented through a solution to a PDE. This study extends previous findings obtained under a single fixed probability framework to the G-expectation context.

We propose a new risk sensitive reinforcement learning approach for the dynamic hedging of options. The approach focuses on the minimization of the tail risk of the final P&L of the seller of an option. Different from most existing reinforcement learning approaches that require a parametric model of the underlying asset, our approach can learn the optimal hedging strategy directly from the historical market data without specifying a parametric model; in addition, the learned optimal hedging strategy is contract-unified, i.e., it applies to different options contracts with different initial underlying prices, strike prices, and maturities. Our approach extends existing reinforcement learning methods by learning the tail risk measures of the final hedging P&L and the optimal hedging strategy at the same time. We carry out comprehensive empirical study to show that, in the out-of-sample tests, the proposed reinforcement learning hedging strategy can obtain statistically significantly lower tail risk and higher mean of the final P&L than delta hedging and other local hedging methods. This is a joint work with Bo Xiao and Xiang Zhou at the City University of Hong Kong and Yi Wu at Peking University.

Financial scenario simulation is essential for risk management and portfolio optimization, yet it remains challenging especially in high-dimensional and small data settings common in finance. We propose a diffusion factor model that integrates latent factor structure into generative diffusion processes, bridging econometrics with modern generative AI to address the challenges of the curse of dimensionality and data scarcity in financial simulation. By exploiting the low-dimensional factor structure inherent in asset returns, we decompose the score function---a key component in diffusion models---using time-varying orthogonal projections, and this decomposition is incorporated into the design of neural network architectures. We derive rigorous statistical guarantees, establishing non-asymptotic error bounds for both score estimation at $O(d^{5/2}n^{-\frac{2}{k+5}})$ and generated distribution at $O(d^{5/4}n^{-\frac{1}{2(k+5)}})$, primarily driven by the intrinsic factor dimension $k$ rather than the number of assets $d$, surpassing the dimension-dependent limits in the classical nonparametric statistics literature and making the framework viable for markets with thousands of assets. Numerical studies confirm superior performance in latent subspace recovery under small data regimes. Empirical analysis demonstrates the economic significance of our framework in constructing mean-variance optimal portfolios and factor portfolios. This work presents the first theoretical integration of factor structure with diffusion models, offering a principled approach for high-dimensional financial simulation with limited data. This is joint work with Minshuo Chen (Northwestern), Renyuan Xu (NYU), and Yumin Xu (PKU).

We consider the problem of optimal multiple stopping where the stopping times should satisfy some distribution constraints. For a large class of cost functions, we reformulate the problem into a sequence of optimal stopping problem of a time-reversed process, explicitly construct the solution and verify the optimality using a martingale inequality. The methodology has links with the Root’s solution to Skorokhod embedding problem and the inverse boundary hitting problem.

We investigate a framework for identifying an agent's risk aversion through interactive questioning. First, we study a one-period setting where the agent's risk aversion is characterized by a state-dependent cost function and a distortion risk measure. We establish the quantitative identifiability of this framework, proving that a finite number of interactions suffices to estimate the true risk aversion within a specified accuracy. Next, we analyze question design efficiency to accelerate estimation and derive a theoretical upper bound on convergence. We propose a novel design method based on distinguishing power and evaluate its performance via simulations. Additionally, we extend our analysis to an infinite-horizon setting, incorporating a discount factor to model dynamic risk aversion. Our approach to inferring risk preferences enables personalized robo-advising tailored to individual clients' needs. This is based on a joint work with Anthony Coache (Imperial) and Sebastian Jaimungal (UToronto).

The focus of this talk is the transformation of increments of a stochastic process by a predictable function. Many operations in stochastic analysis can be considered under this point of view. Stochastic integrals, for example, are linear functionals of process increments. Although mathematically equivalent, focusing on transformation of increments often leads to simpler proofs of more general statements in stochastic calculus. In this talk specifically, we illustrate how considering variations lead to various Ito-type formulas. Joint work with Ales Cerny.

In this talk I will discuss several models of interacting particle systems with absorption. These models arise naturally in finance where a limited resource is available to control potential risks. In the first part of this talk, I will discuss the 'Up the River' problem where a unit drift is distributed among a finite collection of Brownian particles on the positive real line, which are annihilated once they reach the origin. The analysis relies on the hydrodynamic approach and rank-dependent SDEs. This is based on joint work with Li-Cheng Tsai. In the second part, I will talk about McKean-Vlasov equations involving hitting times. This is motivated from modeling systemic risk in the financial market, and relies on tools from partial differential equations. Joint work with Erhan Bayraktar, Gaoyue Guo and Paul Yuming Zhang.

This talk studies a discrete-time mean-variance model based on reinforcement learning. Compared with its continuous-time counterpart in \cite{zhou2020mv}, the discrete-time model makes more general assumptions about the asset's return distribution. Using entropy to measure the cost of exploration, we derive the optimal investment strategy, whose density function is also Gaussian type. Additionally, we design the corresponding reinforcement learning algorithm. Both simulation experiments and empirical analysis indicate that our discrete-time model exhibits better applicability when analyzing real-world data than the continuous-time model.

We investigate a leader-follower game in which the leader hires the follower to complete a project with the presence of a random shock time. If the project is completed before the shock time, then both players receive (up to discounting) $1 each. If it is completed after the shock time, then the leader and the follower receive $y and $x respectively. The shock time is observable by the leader, but not by the follower. The leader chooses how to reveal the information of the shock time, and the follower controls the effort level which affects the project completion time. The goal is to find the leader’s value and optimal information disclosure strategy. By considering the leader’s value as a function of the follower’s utility as well as the follower’s belief about the shock time, we characterize the leader’s value using dynamic programming equations. The leader’s (ε-)optimal strategy can also be constructed from these equations.

In this talk, we present our results on the study of mean-field reflected backward stochastic differential equations (MF-RBSDEs) driven by a marked point process and MF-RBSDEs driven by a Poisson process. Based on a g-expectation representation lemma, we establish the existence and uniqueness of the particle system of MF-RBSDEs driven by a marked point process under Lipschitz generator conditions and obtain a convergence result of this system. In the Poisson setting, we obtain furthermore the convergence rate of the corresponding particle system toward the solution to the MF-RBSDEs driven by a Poisson process under bounded terminals and bounded obstacle conditions.

It is well known that the global (in time) well-posedness of mean field game master equations relies on certain monotonicity conditions, and there have been several types of conditions proposed in the literature. In this talk we intend to provide a unified understanding on the role of monotonicity conditions in the theory. Inspired by Lyapunov functions for dynamical systems, we propose a general type of monotonicity condition, which covers all the existing ones as special cases and is essentially necessary for the existence of Lipschitz continuous classical solutions. Our approach works for very general mean field games, including extended mean field games and mean field games with volatility control. In particular, for the latter a new notion of second order monotonicity condition is required. The talk is based on some ongoing joint works with Jianfeng Zhang and Jianjun Zhou.

In an economy with random growth, several long-lived agents with heterogeneous risk-aversions, time-preferences, and income streams make consumption and investment decisions, trading stocks and a long-term bond, and borrowing from and lending to each other. We find in closed form equilibrium stock prices, interest rates, consumption, and trading policies. Agents do not trade stocks, although their returns are time-varying and predictable. Agents dynamically trade the long-term bond in response to growth shocks.

This paper studies the continuous-time q-learning in mean-field jump diffusion models when the population distribution is not directly observable. We propose the integrated q-function in decoupled form (decoupled Iq-function) from the representative agent’s perspective and establish its martingale characterization, which provides a unified policy evaluation rule for both mean-field game (MFG) and mean f ield control (MFC) problems. Moreover, we consider the learning procedure where the representative agent updates the population distribution based on his own state values. Depending on the task to solve the MFG or MFC problem, we can employ the decoupled Iq-function differently to characterize the mean-field equilibrium policy or the mean-field optimal policy respectively. Based on these theoretical findings, we devise a unified q-learning algorithm for both MFG and MFC problems by utilizing test policies and the averaged martingale orthogonality condition. For several financial applications in the jump-diffusion setting, we obtain the exact parameterization of the decoupled Iq-functions and the value functions, and illustrate our q-learning algorithm with satisfactory performance.

This paper contributes to the compactification approach to tackle mean-field control (MFC) problems with Poissonian common noise. To overcome the lack of compactness and continuity issues due to common noise, we exploit the point process representation of the Poisson random measure with finite intensity and propose a pathwise formulation by freezing a sample path of the common noise. We first study a pathwise relaxed control problem in an auxiliary setup without common noise but with finite deterministic jumping times over the finite horizon. By employing the compactification argument for the pathwise relaxed control problem with Skorokhod topology, we establish the existence of optimal controls in the pathwise formulation. To address the original problem, the main challenge is to close the gap between the problem in the original model with common noise and the pathwise formulation. With the help of concatenation techniques over the sequence of deterministic jumping times, we develop a new tool, also interpreted as the superposition principle in the pathwise formulation, to draw a relationship between the pathwise relaxed control problem and the pathwise measure-valued control problem associated to Fokker-Planck equation. As a result, we can bridge the desired equivalence among different problem formulations. We also extend the methodology to solve mean-field games with Poissonian common noise, confirming the existence of a strong mean field equilibrium.

In this talk, we provide a framework to address extensions of Principal–(multi-)agent problems in contract theory in the context of moral hazard. These include: (i) incorporating constraints on the terminal payment $\xi$, such as $\xi = g(X_T)$, $\xi \leq R$; (ii) equal terminal payments in a Principal–multi-agent problem; (iii) settings in which both the principal and the agent act on the system in a competitive, i.e., Nash, fashion. Building upon previous results for Stackelberg games, one is able to reformulate the Principal’s unconventional problem as a standard optimal stochastic control problem, yet with stochastic target constraints. Crucially, in the previous scenarii the target sets are described by level set equations of suitable—application dependent—functions $\Psi$. Leveraging the stochastic representation of level set equations, we extend the aforementioned ideas and show that the optimal strategies and the value of the Principal’s reformulated problem can then be obtained by solving a well-specified system of Hamilton-Jacobi-Bellman equations. We will illustrate our results in the above scenarii. This is a joint work with Camilo Hernández, Nicolás Hernández-Santibáñez, and Emma Hubert.

We examine climate-related exposure within a large credit portfolio, addressing both transition and physical risks. We propose a modeling methodology that begins with the Shared Socioeconomic Pathways (SSP) scenarios and ends with a quantitative description of portfolio losses. The SSP scenarios affect each obligor's physical risk via a DICE-inspired damage function, and their transition risk through production dynamics requiring optimal adjustment. Each obligor determines an optimal energy mix to align its greenhouse gas (GHG) emission trajectory with the SSP targets, while incorporating uncertainties in consumption trajectories. The resulting Gaussian factor model has a dimension comparable to the number of obligors. We introduce two efficient dimension reduction techniques: Polynomial Chaos Expansion and Principal Component Analysis, to enable fast and accurate loss analysis. This is a joint work with Florian Bourgey and Emmanuel Gobet.

In this talk, we establish equilibrium in the presence of heterogeneous information. In particular, there is an insider who receives a private signal, an uninformed agent with no private signal, and a noise trader with semi price-inelastic demand. The novelty is that we allow the insider to decide (optimally) when to acquire the private signal. This endogenizes the entry time and stands in contrast to the existing literature which assumes the signal is received at the beginning of the period. Allowing for optimal entry also enables us to study what happens before the insider enters with private information, and how the possibility for future information acquisition both affects current asset prices and creates demand for information related derivatives. Results are valid in continuous time, when the private signal is a noisy version of the assets’ terminal payoff, and when the quality of the signal depends on the entry time.

We present a novel on-policy algorithm for solving stochastic optimal control (SOC) problems. By leveraging the Girsanov theorem, our method directly computes on-policy gradients of the SOC objective without expensive backpropagation through stochastic differential equations or adjoint problem solutions. This approach significantly accelerates the optimization of neural network control policies while scaling efficiently to high-dimensional problems and long time horizons. We evaluate our method on classical SOC benchmarks as well as applications to sampling from unnormalized distributions via Schrödinger-Föllmer processes and fine-tuning pre-trained diffusion models. Experimental results demonstrate substantial improvements in both computational speed and memory efficiency compared to existing approaches. Joint work with Mengjian Hua and Eric Vanden-Eijnden.

Estimating a d-dimensional distribution μ by the empirical measure μ^n of its samples is an important task in probability theory, statistics and machine learning. It is well known that E[W_p(μ^n,μ)]2p, where W_p denotes the p-Wasserstein metric. An effective tool to combat this curse of dimensionality is the smooth Wasserstein distance W^(σ)_p, which measures the distance between two probability measures after having convolved them with isotropic Gaussian noise N(0,σ^2I). In this paper we apply this smoothing technique to the adapted Wasserstein distance. We show that the smooth adapted Wasserstein distance AW^(σ)_p achieves the fast rate of convergence E[AW^(σ)_p(μ^n,μ)] (Lecture Hall - Academic island（定山院士岛报告厅))

We consider an agent holding a position in a perpetual contract who wishes to liquidate their position by a terminal time $T$. We propose a model which captures the dynamics of the underlying spot price, random fluctuations in the perpetual price, temporary and permanent price impact effects of trading, and the change in wealth due to the funding rate of the perpetual contract. When the funding rate is a linear function of the underlying spot price we solve for the optimal trading strategy in closed form and investigate its frictionless limit. When the funding rate is a non-linear function of the spot price we derive various approximations for the optimal strategy which are applicable when the funding rate or terminal time are small. When the terminal time is small, an approximately optimal strategy can be written as a modification of the closed form strategy with a linear funding rate.

Stochastic portfolio theory (SPT) is a powerful framework for portfolio selection introduced by Robert Fernholz, which is well suited for tackling questions related to outperformance of a buy-and-hold benchmark, such as the market portfolio. Many theoretical and some empirical studies have obtained performance guarantees for a class of functionally generated portfolios in the frictionless setting, but must less is understood in markets with frictions. Here we introduce modern price impact models to SPT, which are broad enough to allow for nonlinear impact and impact decay and can handle semimartingale trading strategies. In this framework we obtain a master formula for additive functional generation of trading strategies, generalizing the celebrated formula from frictionless SPT. As an application, we obtain first results on relative arbitrage with respect to the market portfolio in the price impact setting. Under similar assumptions to the frictionless case, relative arbitrage can be achieved, albeit over a longer time horizon and with a reduced trading intensity. Some numerical simulations will also be presented, time permitting. 

We study a Stackelberg variant of the classical Dynkin game in discrete time, where the​ two players are no longer on equal footing. Player 1 (the leader) announces their stopping strategy first, and Player 2 (the follower) responds optimally. This Stackelberg stopping game can be viewed as an optimal control problem for the leader. Our primary focus is on the time-inconsistency that arises from the leader–follower structure. We begin by using a finite-horizon example to clarify key concepts: the precommitment strategy and equilibrium strategy in the Stackelberg setting, as well as the Nash equilibrium in the standard Dynkin game. We then turn to the infinite-horizon case and study randomized precommitment and equilibrium strategies. We provide a characterization of the optimal precommitment strategy and show that it may fail to attain the supremum. Moreover, we construct a counterexample to demonstrate that a randomized equilibrium strategy may not exist. To address this, we introduce an entropy-regularized Stackelberg stopping game, in which the follower’s optimization is regularized with an entropy term. This modification yields a continuous best response and ensures the existence of a relaxed randomized equilibrium strategy, which can be viewed as an approximation of the exact equilibrium.

Decentralized exchanges (DEXs) are alternative venues to centralized exchanges to trade cryptocurrencies (CEXs) and have become increasingly popular. An arbitrage opportunity arises when the exchange rate of two cryptocurrencies in a DEX differs from that in a CEX. Arbitrageurs can then trade on the DEX and CEX to make a profit. Trading on the DEX incurs a gas fee, which determines the priority of the trade being executed. We study a gas-fee competition game between two arbitrageurs who maximize their expected profit from trading. We derive the unique symmetric mixed Nash equilibrium and find that (i) the arbitrageurs may choose not to trade when the arbitrage opportunity is small; (ii) the probability of the arbitrageurs choosing a higher gas fee is lower; (iii) the arbitrageurs pay a higher gas fee and trade more when the arbitrage opportunity becomes larger and when liquidity becomes higher. The above findings are consistent with our empirical study.

This paper constructs a characterization framework for the minimal entropy martingale measure (MEMM) in stochastic local volatility (SLV) models. Employing martingale duality principles, we derive the governing nonlinear PDE for measure transformations and obtain closed-form solutions for key volatility specifications. A verification theorem with polynomial growth estimates for parabolic systems rigorously establishes the MEMM as generated by our PDE solutions. Analyses of localized/classical Stein-Stein, Heston, and reversionary Heston models reveal the universal entropy-minimizing structure while maintaining analytical tractability of original models. Our results systematically generalize the classical stochastic volatility-based MEMM theory to SLV environments, providing theoretical foundations for derivative pricing in incomplete markets.

We study mean field portfolio games under Epstein-Zin preferences, which naturally encompass the classical time-additive power utility as a special case. In a general non-Markovian framework, we establish a uniqueness result by proving a one-to-one correspondence between Nash equilibria and the solutions to a class of BSDEs. A key ingredient in our approach is a necessary stochastic maximum principle tailored to Epstein-Zin utility and a nonlinear transformation. In the deterministic setting, we further derive an explicit closed-form solution for the equilibrium investment and consumption policies.

We propose a two-timescale evolutionary game approach to solving multi-agent reinforcement learning (MARL) problems. The algorithm design is built on three key components: (1) agents' policies are updated using a perturbed best response, (2) agents’ beliefs about their opponents are updated using the fictitious play rule, and (3) policies and beliefs are updated at different learning rates than those used for Q-value updates. Our approach provably converges to epsilon-Nash equilibria for general-sum MARL problems, without requiring the restrictive assumptions commonly found in the literature. Additionally, we explore applications of our method in areas such as algorithmic collusion and mean-field-type games. This talk is based on joint works with Mathieu Lauriere (NYU Shanghai), Yumin Xu (Peking University), Ruixun Zhang (Peking University), and Mingyue Zhong (CUHK).

Firms shall be able to respond to their competitors’ strategies over time. Back and Paulsen (2009) thus advocate using closed-loop equilibria to analyze classic real-option exercise games but point out difficulties in defining closed-loop equilibria and character- izing the solution. We define closed-loop equilibria and derive a continuum of them in closed form. These equilibria feature either linear or nonlinear investment thresholds. In all closed-loop equilibria, firms invest faster than in the open-loop equilibrium of Grenadier (2002). We confirm Back and Paulsen (2009)’s conjecture that their closed- loop equilibrium (with a perfectly competitive outcome) is the one with the fastest investment and in all other closed-loop equilibria firms earn strictly positive profits. This work is jointly with Zhaoli Jiang and Neng Wang.

This is a preliminary presentation of some ongoing research on an optimal investment-consumption problem over an infinite horizon. We consider an investor exhibiting shortfall aversion, whose preferences are modeled using Epstein-Zin power utility. Specifically, we solve the case with a zero interest rate, where the elasticity of intertemporal substitution (EIS, denoted by ψ) lies in the interval (0, 1), and the product of the relative risk aversion γ and ψ also falls within (0, 1). We analyze the associated differential equations to investigate the properties of their solutions, as well as the existence and uniqueness of a candidate solution to the optimal control problem. The verification of the proposed solution is underway.

We investigate a time-inconsistent mean-field game (MFG) in a Markov chain setting. We first present a classic equilibrium for the MFG and its associated existence result. This classic equilibrium aligns with the conventional equilibrium concept studied in MFG literature when the context is time-consistent. Then we demonstrate that while this equilibrium produces an approximate optimal strategy when applied to the related N-agent games, it does so solely in a precommitment sense. To address this limitation, we investigate distribution-dependent equilibrium and show that such an equilibrium in the MFG can indeed function as an approximate equilibrium in the N-agent game. This talk is based on a joint work with Erhan Bayraktar.

Quantile aggregation with dependence uncertainty has a long history in probability theory, with wide applications in finance, risk management, statistics, and operations research. Using a recent result on inf-convolution of quantile-based risk measures, we establish new analytical bounds for quantile aggregation, which we call convolution bounds. Convolution bounds both unify every analytical result available in quantile aggregation and enlighten our understanding of these methods. These bounds are the best available in general. Moreover, convolution bounds are easy to compute, and we show that they are sharp in many relevant cases. They also allow for interpretability on the extremal dependence structure. The results directly lead to bounds on the distribution of the sum of random variables with arbitrary dependence. We discuss relevant applications in risk management and economics. This joint work is with Jose Blanchet, Henry Lam and Ruodu Wang.

Stochastic policies are widely used in continuous-time reinforcement learning algorithms. However, executing a stochastic policy and evaluating its performance in a continuous-time environment remain open challenges. This work introduces and rigorously analyzes a policy execution framework that samples actions from a stochastic policy at discrete time points and implements them as piecewise constant controls. We prove that as the sampling mesh size tends to zero, the controlled state process converges weakly to the dynamics with coefficients aggregated according to the stochastic policy. We explicitly quantify the convergence rate based on the regularity of the coefficients and establish an optimal first-order convergence rate for sufficiently regular coefficients. Additionally, we show that the same convergence rates hold with high probability concerning the sampling noise and establish a $1/2$-order almost sure convergence when the volatility is not controlled. Building on these results, we analyze the bias and variance of various policy evaluation and policy gradient estimators based on discrete-time observations. Our results provide theoretical justification for the exploratory stochastic control framework in Wang et. al (2020, Journal of Machine Learning Research). This is a joint work with Du Ouyang and Yufei Zhang.

In this talk, we present a new mechanism for approximating Nash equilibria in ergodic mean field games, under the assumptions that the game is both potential and monotone. Drawing inspiration from fictitious play in MFGs and self-interacting dynamics used to approximate the long-time behavior of McKean–Vlasov equations, we introduce a novel algorithm, which we call self-fictitious play. We will outline how coupling methods and the Lions–Lasry divergence can be employed to establish the convergence of this algorithm.