CulturALL Benchmark Introduced to Test LLMs on Multilingual and Multicultural Grounded Tasks

A new benchmark called CulturALL has been developed to evaluate the multilingual and multicultural competence of large language models on grounded tasks, addressing a gap left by existing benchmarks that focus on generic language understanding or superficial cultural trivia. Built through a human-AI collaborative framework, the benchmark involves expert annotators who ensure appropriate difficulty and factual accuracy while LLMs help reduce manual workload. CulturALL incorporates diverse sources to ensure comprehensive scenario coverage, with each item carefully designed to present a high level of difficulty. The benchmark contains 2,610 samples across 14 languages, aiming to assess how models reason within real-world, context-rich scenarios. This initiative responds to the global deployment of LLMs and the need for more sophisticated evaluation tools that go beyond basic language capabilities. The benchmark's development highlights the growing importance of testing AI systems in culturally and linguistically diverse contexts, moving beyond simple trivia to complex, grounded reasoning tasks.