DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation
A new arXiv paper (2605.15532) reveals that up to 69% of prompts in standard chart/document reasoning datasets are zero-delta, meaning teacher and student VLMs produce identical answer distributions, providing minimal learning signal. The authors propose selecting prompts based on answer divergence to expose functional capability gaps.
Key facts
- arXiv paper 2605.15532
- Up to 69% of prompts in standard chart/document reasoning datasets are zero-delta
- Zero-delta prompts cause student improvement to saturate regardless of data scale
- Proposes selecting prompts based on answer divergence (Δ)
- Non-zero divergence is critical for effective distillation
Entities
Institutions
- arXiv