Safe Equilibrium Policy Optimization for Strategic Language Agents
A novel reinforcement learning technique known as Safe Equilibrium Policy Optimization (SEPO) tackles strategic failure modes found in language models. By enhancing expected payoff with penalties for risks associated with exploitability, collusion, and externality costs, SEPO serves as a reward signal for Group Relative Policy Optimization (GRPO). This method was applied to Gemma 4 E4B-it and Qwen 3.5-4B following supervised fine-tuning. In evaluations spanning five strategic scenarios—Iterated Prisoner's Dilemma, repeated auctions, two negotiation types, and Kuhn Poker—SEPO successfully eliminates exploit-pool advantages, reducing the risks of taking advantage of weaker opponents, harmful coordination, and externalized costs. The approach aims to address the connection between strategic failure modes and the language interface itself.
Key facts
- SEPO augments expected payoff with penalties for exploitability, collusion risk, and externality cost.
- Implemented as a reward signal for Group Relative Policy Optimization (GRPO).
- Applied to Gemma 4 E4B-it and Qwen 3.5-4B after supervised fine-tuning.
- Evaluated across Iterated Prisoner's Dilemma, repeated auctions, two negotiation variants, and Kuhn Poker.
- Achieves zero exploit-pool advantage.
- Addresses strategic failure modes in language models fine-tuned with reinforcement learning.
- Targets exploitation of weaker opponents, coordination on harmful equilibria, and cost externalization.
- Method proposed in arXiv paper 2605.30854.
Entities
Institutions
- arXiv