MORPHOGEN Benchmark Tests Gender-Aware Morphological Generation in Multilingual LLMs

A newly introduced benchmark dataset, MORPHOGEN, assesses the proficiency of multilingual large language models in managing grammatical gender and morphological agreement across three linguistically diverse languages: Hindi, Arabic, and French. Researchers developed a comprehensive synthetic dataset to evaluate 15 widely used multilingual LLMs, which vary in size from 2B to 70B parameters. The primary task, GENFORM, challenges models to transform a first-person sentence into the opposite gender while maintaining its meaning and structure. Although multilingual LLMs excel at tasks such as translation and question answering, their handling of grammatical gender—impacting verb conjugation, pronouns, and first-person forms—has not been thoroughly examined. This study uncovers notable deficiencies in gender-aware morphological generation capabilities. The dataset was published on arXiv under identifier 2604.18914v1 as a type of cross-announcement.