SDGBiasBench: New Benchmark Reveals SDG Biases in Vision-Language Models

A new benchmark suite, SDGBiasBench, has been launched by researchers to assess biases in Vision-Language Models (VLMs) concerning the Sustainable Development Goals (SDGs). This extensive benchmark includes 500,000 multiple-choice questions crafted with expert input and 50,000 regression tasks, allowing for a thorough evaluation of biases at both decision and estimation levels. Findings from SDGBiasBench indicate a fundamental SDG bias in existing VLMs, where predictions are shaped by the inadequate use and integration of evidence. This research fills a void in current benchmarks that often examine qualitative and quantitative aspects separately, which can obscure systematic biases arising from models relying on prior assumptions instead of evidence. The study is available on arXiv titled "SDGBiasBench: Benchmarking and Mitigating Vision--Language Models' Biases in Sustainable Development Goals" (arXiv:2605.21919).