VisDeception Benchmark Tests VLM Robustness to Misleading Charts
A team of researchers has launched VisDeception, the inaugural controlled paired benchmark designed to assess the resilience of Vision-Language Models (VLMs) against misleading chart designs. This benchmark features 1,600 pairs of accurate and deceptive charts across eight primary categories of misleading visualization techniques, such as inverted or truncated axes, distorted aspect ratios, improper encodings, and deceptive color schemes. Each deceptive chart is matched with a truthful version derived from the same data set. This initiative seeks to evaluate the susceptibility of VLMs to such misleading visualizations, which is essential for reliable data analysis, as VLMs are increasingly employed for interpreting charts and conducting analytical reasoning. The benchmark effectively separates reasoning errors caused by deception from other influences.
Key facts
- VisDeception is the first controlled paired benchmark for VLM robustness to misleading charts.
- The benchmark contains 1,600 paired faithful and misleading charts.
- Eight major categories of deceptive visualization tactics are covered.
- Deceptive tactics include truncated axes, inverted axes, distorted aspect ratios, inappropriate encodings, and misleading color mappings.
- Each misleading chart is paired with a faithful counterpart from the same data.
- The study focuses on VLMs used for chart understanding and analytical reasoning.
- The benchmark isolates deception-induced reasoning errors.
- The research is published on arXiv under identifier 2607.22600.
Entities
Institutions
- arXiv