<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="samarskyi-bojar-2026-reverse">
    <titleInfo>
        <title>Reverse Engineering the Reversal Curse: What Entity Order in Pretraining Does and Does Not Explain</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ivan</namePart>
        <namePart type="family">Samarskyi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ondřej</namePart>
        <namePart type="family">Bojar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dana</namePart>
            <namePart type="family">Arad</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yonatan</namePart>
            <namePart type="family">Belinkov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hanjie</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nils</namePart>
            <namePart type="family">Feldhus</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aaron</namePart>
            <namePart type="family">Mueller</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hosein</namePart>
            <namePart type="family">Mohebbi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gabriele</namePart>
            <namePart type="family">Sarti</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Martin</namePart>
            <namePart type="family">Tutek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-476-7</identifier>
    </relatedItem>
    <abstract>A causal large language model can retrieve the same fact more confidently in one direction than in the other. This asymmetry is known as the Reversal Curse. The usual explanation links it to next-token prediction: seeing “A is B” trains the model to predict B from A, but not A from B. Prior work supports this with controlled finetuning on synthetic facts. We ask whether the same account still holds in a real pretraining corpus, where facts appear many times, in different contexts, and in both orders. To test this, we work with OLMo 3 7B Base and Dolma, its 5.93-trillion-token stage-1 corpus. For 100 curated relation instances, we locate passages where both entities appear, have two LLM judges check which passages state the expected relation, and probe the model in both directions with contrastive likelihood scores. The raw specific-prompt forward-minus-reverse gap is small and uncertain (0.119, 95% CI [-0.228,0.486]). Once the generic-prompt gap is subtracted, relation naming helps reverse retrieval more than forward (Δ^rel=-0.467, [-0.673,-0.266]) on average, though not uniformly across families. That gap is not predicted by local entity order after we control for exposure, proximity, entity frequency, and relation family.</abstract>
    <identifier type="citekey">samarskyi-bojar-2026-reverse</identifier>
    <location>
        <url>https://aclanthology.org/2026.blackboxnlp-1.23/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>374</start>
            <end>383</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Reverse Engineering the Reversal Curse: What Entity Order in Pretraining Does and Does Not Explain
%A Samarskyi, Ivan
%A Bojar, Ondřej
%Y Arad, Dana
%Y Belinkov, Yonatan
%Y Chen, Hanjie
%Y Feldhus, Nils
%Y Mueller, Aaron
%Y Mohebbi, Hosein
%Y Sarti, Gabriele
%Y Tutek, Martin
%S Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-476-7
%F samarskyi-bojar-2026-reverse
%X A causal large language model can retrieve the same fact more confidently in one direction than in the other. This asymmetry is known as the Reversal Curse. The usual explanation links it to next-token prediction: seeing “A is B” trains the model to predict B from A, but not A from B. Prior work supports this with controlled finetuning on synthetic facts. We ask whether the same account still holds in a real pretraining corpus, where facts appear many times, in different contexts, and in both orders. To test this, we work with OLMo 3 7B Base and Dolma, its 5.93-trillion-token stage-1 corpus. For 100 curated relation instances, we locate passages where both entities appear, have two LLM judges check which passages state the expected relation, and probe the model in both directions with contrastive likelihood scores. The raw specific-prompt forward-minus-reverse gap is small and uncertain (0.119, 95% CI [-0.228,0.486]). Once the generic-prompt gap is subtracted, relation naming helps reverse retrieval more than forward (Δ^rel=-0.467, [-0.673,-0.266]) on average, though not uniformly across families. That gap is not predicted by local entity order after we control for exposure, proximity, entity frequency, and relation family.
%U https://aclanthology.org/2026.blackboxnlp-1.23/
%P 374-383

Markdown (Informal)

[Reverse Engineering the Reversal Curse: What Entity Order in Pretraining Does and Does Not Explain](https://aclanthology.org/2026.blackboxnlp-1.23/) (Samarskyi & Bojar, BlackboxNLP 2026)

• Reverse Engineering the Reversal Curse: What Entity Order in Pretraining Does and Does Not Explain (Samarskyi & Bojar, BlackboxNLP 2026)

ACL

• Ivan Samarskyi and Ondřej Bojar. 2026. Reverse Engineering the Reversal Curse: What Entity Order in Pretraining Does and Does Not Explain. In Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 374–383, Budapest, Hungary. Association for Computational Linguistics.