<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ulesov-etal-2026-llms">
    <titleInfo>
        <title>Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Makar</namePart>
        <namePart type="family">Ulesov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vladislav</namePart>
        <namePart type="family">Smirnov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Omar</namePart>
        <namePart type="family">Ibrahim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Arsenii</namePart>
        <namePart type="family">Bobovnikov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Jimin</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yongjae</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sophia</namePart>
            <namePart type="family">Ananiadou</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hsin-Hsi</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hiroya</namePart>
            <namePart type="family">Takamura</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Chung-Chi</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-466-8</identifier>
    </relatedItem>
    <abstract>LLM backtest auditing is a calibration problem: a useful auditor must identify methodology flaws without falsely accusing matched clean strategies. We construct a 96-item paired benchmark in which every flawed backtest has a clean control that holds the strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. The benchmark covers eight flaw classes, three difficulty strata, prose and code surfaces, and 1,440 cached audits from four smoke-tested text endpoints. A deterministic scorer separately evaluates flaw classification, clean-control false positives, evidence localization, and fix relevance. The primary auditor reaches 100.0% closed and clean-aware code recall, yet open prompts falsely flag 93.8% of clean code controls. A clean-aware warning reduces its code false-positive rate from 20.8% to 0.0% while preserving 100.0% recall, although clean-aware all-three specificity remains 87.5% because some proposed fixes are incomplete. Model behavior differs sharply under the same clean-aware prompt: some models produce no clean false positives, while another flags 38/48 clean controls despite retaining high recall. Reporting recall alone would therefore hide large differences in practical audit quality. The benchmark does not certify live trading systems; it isolates whether an LLM can identify a planted flaw without manufacturing one in a clean counterpart.</abstract>
    <identifier type="citekey">ulesov-etal-2026-llms</identifier>
    <location>
        <url>https://aclanthology.org/2026.finnlp-1.4/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>26</start>
            <end>30</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
%A Ulesov, Makar
%A Smirnov, Vladislav
%A Ibrahim, Omar
%A Bobovnikov, Arsenii
%Y Huang, Jimin
%Y Lee, Yongjae
%Y Ananiadou, Sophia
%Y Chen, Hsin-Hsi
%Y Takamura, Hiroya
%Y Chen, Chung-Chi
%S Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-466-8
%F ulesov-etal-2026-llms
%X LLM backtest auditing is a calibration problem: a useful auditor must identify methodology flaws without falsely accusing matched clean strategies. We construct a 96-item paired benchmark in which every flawed backtest has a clean control that holds the strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. The benchmark covers eight flaw classes, three difficulty strata, prose and code surfaces, and 1,440 cached audits from four smoke-tested text endpoints. A deterministic scorer separately evaluates flaw classification, clean-control false positives, evidence localization, and fix relevance. The primary auditor reaches 100.0% closed and clean-aware code recall, yet open prompts falsely flag 93.8% of clean code controls. A clean-aware warning reduces its code false-positive rate from 20.8% to 0.0% while preserving 100.0% recall, although clean-aware all-three specificity remains 87.5% because some proposed fixes are incomplete. Model behavior differs sharply under the same clean-aware prompt: some models produce no clean false positives, while another flags 38/48 clean controls despite retaining high recall. Reporting recall alone would therefore hide large differences in practical audit quality. The benchmark does not certify live trading systems; it isolates whether an LLM can identify a planted flaw without manufacturing one in a clean counterpart.
%U https://aclanthology.org/2026.finnlp-1.4/
%P 26-30

Markdown (Informal)

[Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark](https://aclanthology.org/2026.finnlp-1.4/) (Ulesov et al., FinNLP 2026)

• Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark (Ulesov et al., FinNLP 2026)

ACL

• Makar Ulesov, Vladislav Smirnov, Omar Ibrahim, and Arsenii Bobovnikov. 2026. Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark. In Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing, pages 26–30, Budapest, Hungary. Association for Computational Linguistics.