Makar Ulesov
Author directory2026
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Makar Ulesov | Vladislav Smirnov | Omar Ibrahim | Arsenii Bobovnikov
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Makar Ulesov | Vladislav Smirnov | Omar Ibrahim | Arsenii Bobovnikov
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
LLM backtest auditing is a calibration problem: a useful auditor must identify methodology flaws without falsely accusing matched clean strategies. We construct a 96-item paired benchmark in which every flawed backtest has a clean control that holds the strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. The benchmark covers eight flaw classes, three difficulty strata, prose and code surfaces, and 1,440 cached audits from four smoke-tested text endpoints. A deterministic scorer separately evaluates flaw classification, clean-control false positives, evidence localization, and fix relevance. The primary auditor reaches 100.0% closed and clean-aware code recall, yet open prompts falsely flag 93.8% of clean code controls. A clean-aware warning reduces its code false-positive rate from 20.8% to 0.0% while preserving 100.0% recall, although clean-aware all-three specificity remains 87.5% because some proposed fixes are incomplete. Model behavior differs sharply under the same clean-aware prompt: some models produce no clean false positives, while another flags 38/48 clean controls despite retaining high recall. Reporting recall alone would therefore hide large differences in practical audit quality. The benchmark does not certify live trading systems; it isolates whether an LLM can identify a planted flaw without manufacturing one in a clean counterpart.