TIDE: A New Method to Fix On-Policy Distillation Mismatch in LLMs

A recent paper on arXiv (2608.09836) uncovers a failure mode in on-policy distillation (OPD), an essential aspect of contemporary LLM post-training processes. The researchers highlight a phenomenon termed 'degenerate agreement,' where students take advantage of repetitive cycles to attain nearly perfect token alignment with the teacher, despite overall flawed outputs. They redirect attention from mere agreement to the mismatch between teacher and student, classifying mismatch tokens into two categories: student-excess tokens (produced by the student but given almost zero probability by the teacher, leading to unstable updates) and student-deficit tokens (favored by the teacher but seldom selected by the student, hindering reasoning pattern transfer). To tackle these issues, they introduce TIDE (Token-level Independent Deficit-Excess correction), which implements bounded corrections. The abstract of this newly announced paper on arXiv details both the identified problem and the proposed solution, making it significant for the AI/LLM community, especially those focused on distillation and model training.