BilliardPhys-Bench Tests Physical Reasoning in Multimodal LLMs

BilliardPhys-Bench has been developed by researchers as a benchmark to assess physical reasoning capabilities in multimodal large language models (MLLMs) through synthetic billiards environments. This benchmark features a procedural engine that creates randomized scenarios, incorporating elements like friction and elastic collisions, to evaluate three main skills: predicting interactions between balls, reasoning about bounces off walls, and estimating the final positions of balls after they come to a stop. Recent evaluations of MLLMs, including those from the GPT, Claude, Gemini, and Qwen families, show a decline in performance with longer simulation times and more complex scene geometries. A notable issue is "stasis bias," where models often predict no interaction when the correct outcome is less obvious, underscoring limitations in MLLMs' intuitive physical reasoning amidst strong static image recognition.