VideoGameBench: Testing VLMs on 1990s Video Games

Researchers have introduced VideoGameBench, a benchmark consisting of ten popular video games from the 1990s designed to evaluate vision-language models (VLMs) on tasks like perception, spatial navigation, and memory management. Unlike existing benchmarks that rely on coding or math problems, VideoGameBench requires models to complete entire games using only raw visual inputs and high-level descriptions of objectives and controls. Three of the games are kept secret to encourage generalization. The work is detailed in arXiv paper 2505.18134.