MultiGlobeQA: New Multilingual Benchmark Exposes LLM Gaps in Geospatial Reasoning
A new multilingual benchmark called MultiGlobeQA has been launched by researchers to assess geospatial reasoning in large language models (LLMs). This benchmark features 46,060 question-answer pairs across 14 spatial-function categories and 15 answer types, utilizing execution-based ground truth derived from three knowledge graphs. It includes data from 201 countries and territories, employing income- and density-stratified sampling, and offers parallel questions in English alongside 16 other high- and low-resource languages. According to the study published on arXiv (ID 2608.03882), LLMs face challenges with geometric and topological computations, even though they possess extensive geographic knowledge. The benchmark seeks to deliver a more thorough and controlled evaluation compared to earlier synthetic or limited monolingual benchmarks, emphasizing the necessity for enhanced spatial reasoning in AI, especially for navigation and logistics.
Key facts
- MultiGlobeQA is a multilingual benchmark for geospatial reasoning.
- It contains 46,060 question-answer pairs.
- It spans 14 spatial-function families and 15 answer formats.
- Ground truth is execution-based over three knowledge graphs.
- It covers 201 countries and territories via income- and density-stratified sampling.
- Questions are in English and 16 additional languages.
- LLMs fail on tasks requiring grid indexing.
- The benchmark is introduced in arXiv paper 2608.03882.
Entities
Institutions
- arXiv