GPT-4o and Other Multimodal Models Benchmarked on Standard Vision Tasks

A recent study published on arXiv evaluates several multimodal foundation models (MFMs), such as GPT-4o, o4-mini, Gemini 1.5 Pro, Gemini 2.0 Flash, Claude 3.5 Sonnet, Qwen2-VL, and Llama 3.2, focusing on typical computer vision tasks including semantic segmentation, object detection, image classification, and predicting depth and surface normals. Utilizing well-known datasets like COCO and ImageNet, the research highlights significant challenges, particularly that many models primarily generate text and struggle to represent visual elements like segments or 3D structures. Additionally, numerous models are proprietary and accessible only through APIs. To tackle these issues, the authors propose converting vision tasks into text-promptable formats using prompt chaining, establishing a consistent evaluation framework.