A-3PO: Staleness-Aware Proximal Policy Approximation Speeds Up Asynchronous LLM Training

A new paper on arXiv (ID 2512.06547) introduces A-3PO (APproximated Proximal Policy Optimization), a method to accelerate asynchronous large language model (LLM) training under reinforcement learning (RL). The paper, titled "A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation," addresses the computational overhead in decoupled PPO, a variant of PPO designed for asynchronous RL settings. Decoupled PPO improves learning stability by using a proximal policy to decouple off-policy correction from policy update constraints, but this requires an extra forward pass per training step. A-3PO approximates the proximal policy via simple interpolation, eliminating the overhead and achieving a 1.8x speedup while maintaining performance. The approach is based on the observation that the proximal policy serves only as a trust region anchor between behavior and target policies. The paper is available at https://arxiv.org/abs/2512.06547.