VSSD: Visual Saliency Steering Distillation Improves Multimodal CoT Reasoning
Researchers propose Visual Saliency Steering Distillation (VSSD), a method to enhance multimodal chain-of-thought reasoning in small models. VSSD uses attention maps from multimodal large language models to generate perturbed images capturing task-sensitive feature directions, then applies singular value decomposition to extract steering vectors for inter-layer distillation. Experiments on ScienceQA and M³CoT show improved rationale generation and answer inference. The code is publicly available.
Key facts
- VSSD leverages attention maps of multimodal large language models.
- It generates perturbed images to capture task-sensitive feature directions.
- Singular value decomposition extracts dominant steering vectors.
- Steering vectors guide inter-layer distillation.
- Experiments conducted on ScienceQA and M³CoT datasets.
- VSSD improves rationale generation and answer inference.
- The method addresses token budget limitations in small models.
- Code is available at the provided URL.
Entities
—