New Arabic Cultural QA Benchmark Tests LLMs on Dialect Variants and Open-Ended Questions

A new research method proposes a comprehensive Arabic cultural question-answering benchmark that addresses gaps in large language model performance. The approach translates Modern Standard Arabic multiple-choice questions into English and several Arabic dialects, then converts them into open-ended formats. Researchers benchmarked a range of zero-shot and fine-tuned LLMs under both multiple-choice and open-ended settings. The method also generates chain-of-thought rationales to fine-tune models for step-by-step reasoning. Using this approach, researchers extended an existing dataset where questions and answers are parallelly aligned across multiple language varieties. Extensive experiments were conducted with both open and closed models. Findings reveal that models consistently underperform on Arabic dialects, highlighting persistent gaps in culturally grounded content. The dataset represents what researchers believe to be the first of its kind for Arabic language varieties.