Soyeon Lee
Author directory2026
Beyond Output-Level Correction: Tracing Positional Bias to Early-Layer Commitment in Transformer Multiple-Choice Evaluation
Soyeon Lee | Minhyeok Lee
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Soyeon Lee | Minhyeok Lee
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Language models exhibit positional bias on multiple-choice benchmarks, scoring the same question differently depending on which position holds the correct answer. Existing work measures the bias at the output and corrects it by shuffling option orders, but does not explain where in the network it originates. Using causal activation patching with a formalized Average Indirect Effect (AIE) metric, we trace the layer-by-layer formation of positional commitment across three model families and benchmarks. Positive causal mass concentrates in early-to-mid layers (normalized depth below 0.60) for the higher-capacity models tested, with Pearson correlations above 0.89 across nine model-dataset conditions and three output metrics, and the localization is stable under multi-layer windowed patching. A controlled factorization that permutes option labels and option order independently shows this profile tracks the label-identity component of the bias rather than generic order sensitivity. Sublayer decomposition reveals MLP-dominated early layers (about 59%) and attention-dominated mid-layers (about 76%), consistent with label information being integrated through attention after semantic content is encoded by MLPs. Negative controls show no detectable early-layer separation between robustly correct and bias-manifesting questions, supporting a shared answer-commitment pathway rather than a dedicated bias circuit and explaining why output-level shuffling stabilizes scores without altering internal preference.