Instruction Stacking Collapse: Benchmark Reveals Capability-Dependent Value of Prompt Compilation
A recent study published on arXiv (2608.02639) examines the decline in instruction adherence among large language models as more constraints are introduced. It evaluates three leading LLMs: Claude Sonnet 4.6, GPT-5-mini, and Gemini 2.5 Flash, using up to 20 verified instructions. The findings reveal a significant non-linear drop in adherence rates, plummeting from approximately 96% to as low as 20%, primarily due to structured pairwise conflicts. For example, a single 'output JSON' requirement cannot be satisfied alongside nine others. Additionally, the research explores a training-free solution: an instruction compiler that reformulates stacked prompts in one LLM call, enhancing performance by up to +11 points for weaker models, which face the greatest challenges with stacking. These insights are crucial for prompt engineering and AI system development, especially in scenarios demanding intricate multi-constraint instructions.
Key facts
- Benchmark stacks 24 verifier-checked instructions, one to twenty at a time.
- Evaluates Claude Sonnet 4.6, GPT-5-mini, and Gemini 2.5 Flash.
- Follow rate falls from ~96% to as low as 20%.
- Degradation is non-linear and driven by pairwise conflicts.
- A single 'output JSON' constraint is jointly unsatisfiable with nine others.
- Instruction compiler rewrites stacked prompt in a single LLM call.
- Compiler recovers up to +11 points of follow rate for weaker models.
- Benefit is capability-graded.
Entities
Institutions
- arXiv