New Research Exposes Informativeness Bias in Vision-Language Model Judges

A new study has revealed a serious flaw in using vision-language models (VLMs) to evaluate other VLMs. This research, found on arXiv under the ID 2604.17768v1, shows that these models often miss important details in images when making their assessments, choosing instead to focus on providing informative but potentially misleading answers. This issue, called "informativeness bias," significantly weakens automated evaluation methods. To tackle this challenge, the researchers proposed a new framework named BIRCH (Balanced Informativeness and CoRrectness with a Truthful AnCHor), which aligns candidate responses with actual image content. BIRCH emphasizes accuracy over just being informative, cutting down informativeness bias by up to 17% and boosting performance by as much as 9%. The study highlights critical flaws in current evaluation strategies for VLMs.