Simulator Collapse in Multi-Agent RL: A New Challenge for Human-AI Interaction

A new study published on arXiv (2608.12253) reveals a significant issue in multi-agent reinforcement learning concerning human-AI interactions, termed simulator collapse. The researchers illustrate that depending on a single large language model (LLM) to mimic user behavior results in consistent generalization failures. This issue stems from the mode collapse of the simulator LLM, leading to an LLM policy that overly adapts to limited strategies exploiting the dominant mode of the simulator. Consequently, such policies perform poorly with new simulators and actual users. The paper theoretically defines this collapse and suggests two solutions: Verbalized Sampling, which enhances simulator behavior during inference, and Co-Training, which optimizes policy against multiple trainable simulators. The implications for developing resilient AI systems, especially in chatbots, virtual assistants, and gaming, are profound. The paper can be accessed on arXiv with the identifier 2608.12253.