Explanation Quality Assessment via Ranking Rewards

Researchers have redefined the assessment of explanation quality as a ranking challenge, focusing on training reward models to evaluate various candidate explanations instead of producing a single optimal explanation sequentially. They create sets of candidates for each instance, categorized by quality levels, and employ listwise and pairwise ranking models (ListNet, LambdaRank, RankNet) to maintain the ordinal structure and prevent the score compression often seen in pointwise regression or binary preference tasks. Results indicate that ranking losses consistently surpass regression in terms of score differentiation across all tested domains. The choice of the best ranking loss is influenced by data characteristics: listwise objectives perform well with distinct quality tiers, while pairwise approaches are more resilient to noisy natural annotations. This research utilizes meticulously curated and structured data.