LLM Agents Reproduce Social Science Results from Paper Methods Alone

A recent investigation published on arXiv (2604.21965) examines the ability of LLM agents to replicate findings in social science by utilizing solely the methodology outlined in research papers and the original datasets, without any access to the code or results. The system systematically extracts structured methodologies from the documents, conducts reimplementations while maintaining strict information isolation, and allows for deterministic comparisons at the cell level. An error attribution phase identifies the underlying reasons for any discrepancies. The study, which assesses four agent scaffolds and four LLMs across 48 papers with confirmed human reproducibility, reveals that while agents can generally reproduce published findings, their effectiveness varies significantly among models, scaffolds, and individual papers. Root cause analysis indicates that failures arise from both agent mistakes and additional factors.