TOPPO: Tail-Optimized PPO Reformulates Multi-Task RL via Critic Balancing

A recent study published on arXiv (2605.11473) presents TOPPO (Tail-Optimized PPO), a new take on Proximal Policy Optimization (PPO) tailored for Multi-Task Reinforcement Learning (MTRL). The researchers highlight a critical issue in PPO's application to MTRL: critic-side gradient ill-conditioning, where simpler tasks overshadow value function updates, causing tail tasks to lag. TOPPO counters this with Critic Balancing, a series of modules designed to enhance gradient conditioning and harmonize learning dynamics across various tasks. Unlike earlier methods that depend on modular designs or extensive models, TOPPO focuses on resolving the optimization challenges inherent in PPO. In experiments, TOPPO demonstrates superior mean and tail-task performance compared to established SAC-family and ARS-family benchmarks, while utilizing significantly fewer parameters and environment steps on the Meta-World benchmark. This research was announced as a new submission on May 14, 2025.