Agent Island: Dynamic Benchmark for Language Model Agents

Agent Island is a multiplayer simulation environment where language-model agents compete in games of cooperation, conflict, and persuasion. It serves as a dynamic benchmark designed to resist saturation and contamination, common issues in static benchmarks. New models can always outperform the current leader in this winner-take-all game, as agents face adaptive opponents rather than fixed tasks. Players are ranked using a Bayesian Plackett-Luce model, which quantifies uncertainty in skill. In 999 games involving 49 unique models, openai/gpt-5.5 leads with a posterior mean skill of 5.64, followed by openai/gpt-5.2 at 3.10 and openai/gpt-5.3-codex at 2.86. The game logs are released as a dataset.