COMPINT: Evaluating Session Constraint Loss in Context Compaction

A recent study published on arXiv (2608.11242) uncovers a significant issue in the context compaction of LLMs: Session Constraints (SCs) issued by users, like "do not delete any emails until I confirm," are frequently omitted during the compaction process. The researchers present COMPINT, an evaluation framework designed to assess compactors in three long-context situations: multi-turn dialogue, agentic trajectory, and extended research. Their results indicate that existing compactors preserve merely 17% of the provided SCs on average, with many performing worse than executing the task without compaction. Retention rates fluctuate significantly based on the compactor, prompt, context length, SC wording, and injection point, revealing a systemic problem. To address this, the paper suggests an SC-aware extractor that operates alongside the compactor as a modular solution, emphasizing a critical challenge for LLM systems regarding user oversight and task consistency.