MNPO: Multiplayer Nash Preference Optimization for LLM Alignment

A novel approach known as Multiplayer Nash Preference Optimization (MNPO) has been developed to overcome the shortcomings of current reinforcement learning from human feedback (RLHF) techniques. While RLHF serves as the conventional method for aligning large language models (LLMs) with human preferences, reward-based strategies based on the Bradley-Terry assumption often fail to adequately represent the nontransitivity and diversity of actual preferences. Recent research has reinterpreted alignment as a two-player Nash game, resulting in Nash learning from human feedback (NLHF). Although algorithms like INPO, ONPO, and EGPO provide robust theoretical and empirical support, they are limited to two-player scenarios, creating a single-opponent bias. MNPO extends NLHF to a multiplayer context, framing alignment as an n-player game, with each participant reflecting a unique preference. This framework seeks to more accurately represent the intricacies of real-world preference systems. The paper can be found on arXiv with the identifier 2509.23102, categorized under the announcement type 'replace'.