ICML 2026 Open Reproductions Challenge: 2,226 Papers Audited by Community Agents
The ICML 2026 Open Reproductions challenge, hosted by Hugging Face and alphaXiv, took place from July 15 to August 2, 2026. This initiative saw 1,221 participants utilizing AI coding agents to reproduce conference papers, indexing 6,341 accepted submissions and extracting key claims for verification through tools such as Claude Code and Codex. Logbooks totaling 6,816 were published for 2,226 papers, representing 34% of the conference. An automated Logbook Judge evaluated 35,908 claims, revealing that 51% of the papers (1,103) had at least one verified claim, while 23% (496) contained falsified claims. Significant issues were identified in a paging paper, an attention theorem, a self-distillation paper, and a transformer paper. Authors have responded positively, with two corrections on arXiv pending.
Key facts
- The ICML 2026 Open Reproductions challenge ran from July 15 to August 2, 2026.
- 1,221 community members participated, publishing 6,816 reproduction logbooks.
- 2,226 papers were attempted, covering 34% of the conference's accepted papers.
- 35,908 claims were judged by an automated Logbook Judge using GLM-5.2.
- 51% of examined papers had at least one claim verified; 23% had at least one claim falsified.
- 266 papers were fully reproduced, and 49 papers had all claims falsified.
- Confirmed falsifications included a paging paper with a flawed proof and a theorem failing after step 224.
- Authors confirmed findings and two arXiv corrections are in flight.
- The challenge was organized by Hugging Face and alphaXiv.
- Participants used AI agents like Claude Code, Codex, Cursor, and orx.
Entities
Institutions
- Hugging Face
- alphaXiv
- ICML
- Claude Code
- Codex
- Cursor
- OpenResearch
- GLM-5.2
- arXiv