Reward Bias Substitution: Single-Axis Mitigations Redirect Optimization Pressure

A recent study available on arXiv (2605.27996) uncovers a flaw in the approach to mitigating reward-model bias, termed reward bias substitution. Techniques that focus on a single aspect—like minimizing reliance on length, sycophancy, or style—may shift optimization pressure onto related proxies instead of resolving it. This issue arises from a discrepancy between measurement and optimization during the evaluation of mitigation and the training of policies. The researchers categorize mitigation results into a regime taxonomy and demonstrate that successful mitigation, bias substitution, and overcorrection yield the same observable outcomes across any audit-distribution scoring, including ranking accuracy and win-rate, even with perfect knowledge of the actual reward. A review of existing preference-learning mitigation strategies reveals that none provide the necessary evidence to confirm successful mitigation. Enhancing evaluation by incorporating policy-induced distributions while monitoring multiple biases effectively bridges this gap.