ArabCulture-Dialogue: Benchmarking LLMs on Arabic Cultural Reasoning

Researchers have introduced ArabCulture-Dialogue, a conversational dataset designed to evaluate cultural reasoning in large language models (LLMs) across Arabic dialects and Modern Standard Arabic (MSA). The dataset covers 13 Arabic-speaking countries, includes both MSA and each country's respective dialect, and spans 12 daily-life topics with 54 fine-grained subtopics. Three benchmarking tasks were developed: multiple-choice cultural reasoning, machine translation between MSA and dialects, and dialect-steering generation. Experiments reveal a persistent performance gap between MSA and Arabic dialects, with models performing worse on all three tasks in dialectal setups compared to MSA. The work addresses a significant gap in evaluating cultural nuances in LLMs, as most Arabic benchmarks rely on short text snippets in MSA and overlook conversational contexts.