arXiv Study Reveals Authenticity Gap in Human Evaluation of Natural Language Generation

Human assessments are regarded as the benchmark for assessing natural language generation (NLG) systems; however, an arXiv preprint (ID 2205.11930v3) suggests that this approach frequently misrepresents genuine human preferences. The authors apply utility theory to scrutinize the evaluation protocol, uncovering that underlying assumptions about annotators may distort ratings. They indicate that Likert scales could potentially invert authentic preferences. While they suggest enhancements to the protocol, they acknowledge its limitations in evaluating open-ended tasks, such as story generation. A novel human evaluation methodology is proposed, underscoring the necessity for reliable assessment strategies across diverse tasks. This study raises concerns about the dependability of existing methods and emphasizes the gap between numerical averages and true human evaluations.