Study Finds Evaluation Metrics Fail to Detect Errors in Classical Chinese to English Translations

A recent investigation published on arXiv examines the reliability of automatic evaluation metrics designed for contemporary languages when applied to translations from Classical Chinese to English, a topic of growing importance in digital humanities. The authors present a diagnostic framework utilizing minimal pairs to identify common error types relevant to academic contexts, analyzing both reference-based and reference-free metrics for their sensitivity to errors and tolerance for legitimate variations. Results indicate that all metrics have limitations, with MetricX24 showing the best performance overall. This research underscores the necessity for more effective and interpretable metrics suited for unique historical and cultural translation contexts. The study is part of a larger inquiry into the applicability of large language models in digital humanities, where dependable evaluation remains a significant challenge. The paper is accessible on arXiv under the identifier 2608.08283.