Calibrated Interactive RL Addresses Distribution Shift in LLM Dialogue

A recent study published on arXiv (2605.26403) highlights context distribution shift as a critical challenge in training dialogue agents based on LLMs. The researchers demonstrate that both Static Context RL (which relies on fixed offline logs) and Interactive RL (which utilizes prompt-based simulators) experience a disconnect between training dialogues and actual conversations, leading to a quadratic degradation in quality over multiple turns. They identify two main causes for this shift: one stemming from policy-induced changes due to static histories and the other from simulator-induced discrepancies between human behavior and simulations. To tackle this issue, they introduce Calibrated Interactive RL, a comprehensive framework that integrates interactive RL with a calibrated simulator, aiming to reduce both types of shifts and advance the development of highly interactive LLM agents.