Systematic Evaluation of AI Agents for Long-Horizon Research

A recent paper on arXiv (2608.13417) provides a comprehensive assessment of seven advanced AI models across 36 long-term tasks, utilizing an innovative framework that goes beyond mere final scores to analyze behavior during runs. This framework employs rule-based metrics to evaluate Solution Framing, Execution, and Feedback Control, alongside controlled comparisons to examine experience reuse within and between tasks. Findings reveal that existing agents function more like engineering optimizers than independent researchers; they can devise and execute effective solutions, yet their performance fluctuates significantly across different runs, primarily relying on adapting previous knowledge rather than creating original strategies. The study underscores the necessity for evaluation techniques that reflect the dynamics of long-term experimentation, as final scores fail to indicate areas of progress or setbacks. The unnamed authors of the paper suggest that while AI agents can tackle practical tasks, they do not possess the autonomy and reliability of human researchers. The proposed framework could act as a standard for future assessments, highlighting the significance of process-oriented metrics in evaluating AI capabilities.