Study Reveals Robustness Issues in Multilingual Text Embedding Rankings

A new meta-study from arXiv examines the robustness of multilingual text embedding model rankings across different learning tasks, languages, and benchmark datasets. The research focuses on the MTEB (Massive Text Embedding Benchmark) platform, which evaluates models across over 250 languages. The study introduces two robustness indicators: dataset-composition robustness and ranking-scheme robustness, to assess whether conclusions about model superiority remain stable under varying evaluation designs. The authors apply multi-criteria decision-making ranking schemes to analyze sensitivity to changes in dataset composition and aggregation methods. The findings highlight that implicit choices in benchmark design can significantly affect perceived model performance, urging caution in interpreting leaderboard rankings.