Cross-Temporal Legal NLP Benchmarks Show Severe Performance Decay

A recent study questions the assumption of stationarity in legal NLP by examining transformer models on Ukrainian court rulings from three distinct geopolitical periods: pre-war (2008–2013), hybrid war (2014–2021), and full-scale invasion (2022–2026). Researchers fine-tuned four transformer encoders—XLM-RoBERTa base and large, along with their legal-domain adaptations—on one period and assessed them across all three, resulting in a 3×3 cross-temporal generalization matrix. The findings reveal significant forward degradation, with models trained on pre-war data experiencing a drop of up to 27.2 percentage points in macro-F1 when used on full-scale invasion rulings. In contrast, backward transfer from full-scale to pre-war is notably stronger, supporting the idea that legal language builds upon itself. The enhancement from legal-domain pretraining was minimal compared to general-domain models, highlighting the necessity for temporal awareness in legal AI systems.