David Zollikofer
Author directory2026
Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results
Benjamin Kohler | David Zollikofer | Johanna Einsiedler | Alexander Miserlis Hoyle | Elliott Ash
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Benjamin Kohler | David Zollikofer | Johanna Einsiedler | Alexander Miserlis Hoyle | Elliott Ash
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Recent work uses LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper’s methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation—agents never see the original code, results, or paper—and enables deterministic, cell-level comparison of reproduced outputs to the original results. An LLM-based attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis suggests that failures stem both from agent errors and from underspecification in the papers themselves.