MLLMs' Evidential Basis for Visualization Claims Examined
A study from arXiv investigates how multimodal large language models (MLLMs) generate claims about data visualizations, analyzing 102 charts from four sources using three MLLMs (including Gemini and GPT) under four input conditions. The research, involving 1,224 descriptions, categorizes model outputs as DIRECT, DERIVED, or SPECULATIVE and audits numeric agreement. Accessible chart context increased DIRECT claims and improved numeric agreement for some models, but adding images did not consistently help. The Real-World Significance section remained largely speculative. The study highlights the need for clearer evidential grounding in MLLM-generated interpretations.
Key facts
- Study examines MLLM-generated claims for data visualizations
- 102 visualizations from four sources analyzed
- Three MLLMs tested: Gemini, GPT, and one other
- Four input conditions vary image, context, and framing
- 1,224 descriptions generated and labeled as DIRECT, DERIVED, or SPECULATIVE
- Automated audit of numeric agreement conducted
- Accessible chart context shifted Gemini and GPT toward DIRECT claims
- Adding image to full context did not yield consistent numeric benefit
- Withheld-context prompt did not reliably increase cautious language
- Real-World Significance section remained predominantly SPECULATIVE
Entities
Institutions
- arXiv