Distributional RL Extended to Partially Observable Markov Decision Processes

A recent study published on arXiv advances Distributional Reinforcement Learning (DistRL) to the realm of Partially Observable Markov Decision Processes (POMDPs). The researchers present distributional Bellman operators tailored for partial observability and demonstrate their convergence using the supremum p-Wasserstein metric. They introduce a finite representation of return distributions through psi-vectors, which extend the traditional alpha-vectors found in POMDP solvers. Furthermore, they create Distributional Point-Based Value Iteration (DPBVI), incorporating psi-vectors into a conventional point-based backup method. This research is driven by advancements in world model techniques, where latent models simulate beliefs and facilitate planning. The paper can be found under arXiv:2505.06518v3.