BEAR-Bench: Bilingual Benchmark Tests AI Reasoning on Professional Documents
BEAR-Bench is a newly developed bilingual benchmark aimed at assessing multimodal large language models, featuring 1,000 questions annotated by humans sourced from dense business and scientific texts. Tailored for both English and Russian, it seeks to remedy the limitations of current benchmarks that predominantly focus on English or Chinese and emphasize information extraction. Researchers tested 16 models, both proprietary and open-weight, such as Gemini 3.1 Pro and Qwen3.5-397B, revealing significant potential for improvement, even among the top-performing models. This benchmark is designed to be self-sufficient, requiring no external domain expertise, and strives to address the lack of representation for Russian in AI assessments.
Key facts
- BEAR-Bench is a bilingual benchmark for multimodal large language models.
- It includes 1,000 human-annotated questions.
- Questions are based on text-rich business and scientific documents.
- The benchmark covers English and Russian.
- 16 proprietary and open-weight MLLMs were evaluated.
- Evaluated models include Gemini 3.1 Pro and Qwen3.5-397B.
- Existing benchmarks are largely English- or Chinese-centric.
- Results show headroom even for the strongest systems.
Entities
Institutions
- arXiv