CALM: Training LLMs to Cooperate with Diverse Inference-Time Controllers

A novel post-training approach known as CALM (Controller-Aware Language Models) tackles the discrepancies between the training phase and the deployment of large language models (LLMs). Existing post-training techniques focus on optimizing a singular, fixed interaction style, while actual applications require a range of controllers, including Chain-of-Thought, self-consistency, debate, planning, and verification processes. CALM integrates controllers directly into the training loop, treating controller-aware post-training as multi-task reinforcement learning based on interaction protocols shaped by controllers. These controllers consist of reusable local reasoning modules, allowing for a modular breakdown of mixed-controller training under a turn-level GRPO objective. This methodology enhances adaptability to new workflows and fosters better collaboration with various inference-time controllers.