The Unwritten Benchmark: New AI Challenge Exposes Gap in Abstract Reasoning
A recent research project, titled "The Unwritten Benchmark," explores machines' abilities to grasp abstract concepts through a novel task called acousto-kinematic word inference. This challenge evaluates models based on audio of pen sounds and visual cues from hand movements, avoiding direct exposure to written text. The study reveals that while humans achieve over 80% accuracy with structured letters, advanced AI models like GPT-4o and Gemini 2.5-Pro struggle, scoring below 10%. The findings are significant in underscoring the limitations in artificial intelligence capabilities regarding abstract reasoning and are available on arXiv under ID 2608.14558v1.
Key facts
- The Unwritten Benchmark is a new challenge for multimodal machine learning.
- The core task is acousto-kinematic word inference.
- Models must decipher words from audio of pen scratches and video of hand movements.
- No visible ink trace is provided.
- Words are presented in three different writing styles.
- Human participants achieve over 80% ordered letter accuracy.
- GPT-4o and Gemini 2.5-Pro fail to surpass 10% accuracy.
- The paper is available on arXiv with identifier 2608.14558v1.
Entities
Institutions
- arXiv