Ivan Samarskyi
Author directory2026
Reverse Engineering the Reversal Curse: What Entity Order in Pretraining Does and Does Not Explain
Ivan Samarskyi | Ondřej Bojar
Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
Ivan Samarskyi | Ondřej Bojar
Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
A causal large language model can retrieve the same fact more confidently in one direction than in the other. This asymmetry is known as the Reversal Curse. The usual explanation links it to next-token prediction: seeing “A is B” trains the model to predict B from A, but not A from B. Prior work supports this with controlled finetuning on synthetic facts. We ask whether the same account still holds in a real pretraining corpus, where facts appear many times, in different contexts, and in both orders. To test this, we work with OLMo 3 7B Base and Dolma, its 5.93-trillion-token stage-1 corpus. For 100 curated relation instances, we locate passages where both entities appear, have two LLM judges check which passages state the expected relation, and probe the model in both directions with contrastive likelihood scores. The raw specific-prompt forward-minus-reverse gap is small and uncertain (0.119, 95% CI [-0.228,0.486]). Once the generic-prompt gap is subtracted, relation naming helps reverse retrieval more than forward (𝛥rel=-0.467, [-0.673,-0.266]) on average, though not uniformly across families. That gap is not predicted by local entity order after we control for exposure, proximity, entity frequency, and relation family.