VIB-Probe Framework Aims to Reduce Hallucinations in Vision-Language Models
A new research paper introduces VIB-Probe, a framework designed to detect and mitigate hallucinations in Vision-Language Models. Hallucinations occur when generated text deviates from the actual visual content. The method leverages Variational Information Bottleneck theory to filter out semantic noise while extracting discriminative patterns across model layers and attention heads. Existing detection approaches often rely on output logits or external verification tools, overlooking internal mechanisms. VIB-Probe specifically investigates internal attention heads, postulating that certain heads carry primary signals for truthful generation. Directly probing these high-dimensional states is challenging due to entangled visual-linguistic syntax and noise. The paper is available on arXiv under identifier 2601.05547v2.
Key facts
- VIB-Probe is a hallucination detection and mitigation framework for Vision-Language Models
- It uses Variational Information Bottleneck theory to filter semantic nuisances
- The method extracts discriminative patterns across model layers and attention heads
- Hallucinations refer to generated text deviating from underlying visual content
- Existing methods primarily rely on output logits or external verification tools
- The framework investigates internal attention heads for truthful generation signals
- Direct probing of high-dimensional states is challenging due to syntax-noise entanglement
- The research is documented in arXiv paper 2601.05547v2
Entities
Institutions
- arXiv