New Benchmarks Reveal Cross-Modal Inconsistency in MLLMs

Researchers introduced REST and REST+, two benchmarks for evaluating cross-modal inconsistency in multimodal large language models (MLLMs). These benchmarks contain samples with identical semantic information across image, text, and mixed modalities. Evaluating 15 state-of-the-art MLLMs, the study found substantial variation in modality inconsistency, even when accounting for OCR errors. Neither rendering text as image nor image as text resolved the inconsistency. Visual characteristics like text color and resolution, but not font, and the number of vision tokens affected performance.