Shadow evaluations test AI agents on open-ended research

A new method called shadow evaluations measures AI agents' ability to conduct open-ended AI research. In two case studies, frontier agents were given six days and thousands of dollars of compute to tackle the central research questions of two unpublished NeurIPS 2026 submissions. The original authors graded the output. While agents completed all engineering tasks without human help, they failed to make substantial progress on the research questions. The study introduces shadow evaluations as a third way to measure progress toward AI R&D automation, avoiding limitations of narrow verifiable tasks or unreliable peer review.