Measurement Risk in Financial NLP Benchmarks: Rubric and Metric Sensitivity

A new study from arXiv (2604.27374) investigates measurement risk in supervised financial NLP, focusing on the Japanese Financial Implicit-Commitment Recognition (JF-ICR) benchmark. The research tests four frontier LLMs across five rubrics, three temperatures, and five ordinal metrics on a 253-item test split. Key findings show that rubric wording significantly alters model-assigned labels, with agreement between rubrics ranging from 70.0% to 83.4%. The dominant movement occurs near the +1/0 implicit-commitment boundary, suggesting pragmatic boundary sensitivity. The study challenges the assumption that gold labels provide objective evidence for model selection and deployment, highlighting that benchmark rulers are sensitive to rubric wording, metric choice, and aggregation policy. This work underscores the need for careful benchmark design in financial NLP applications.