As part of the Mathematics Colloquium, Prof. Dr Sergey Foss from Heriot-Watt University in Edinburgh will give a lecture on the topic: "On the instability of local learning algorithms: Q-learning can fail in infinite state spaces".

Date: 15 May 2026

Time: 10.15 am s.t.

Venue: Helmholtzstrasse 18, Room 220

At 9.45 am, there will be a short coffee break on the second floor of HeHo 18, followed by lunch. Please send an email to Carolina Fernandez (carolina.fernandez(at)uni-ulm.de) if you are interested in attending, so that we can draw up a list of participants for the restaurant.

Abstract:

We investigate the challenges of applying model-free reinforcement learning
algorithms, such as online Q-learning, to Markov decision processes (MDPs) with
infinite state spaces. We first introduce the notion of Local Learning Processes
(LLPs), in which agents make decisions based solely on local information, and show
that Q-learning can be viewed as a specific instance of an LLP. Using renewal
techniques, we analyse LLPs and demonstrate their instability under certain drift and
initial conditions, revealing fundamental limitations in infinite state spaces. In
particular, we show that although Q-learning is asymptotically optimal in finite
settings, it can exhibit instability and strict suboptimality in infinite state spaces.