Éric Jacopin
Author directory2026
Present but Barely Used: Reproducing the Emergent Othello World Model in a Masked Diffusion Language Model
Éric Jacopin
Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
Éric Jacopin
Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
Autoregressive (AR) sequence models trained only to predict legal Othello moves develop an emergent, linearly-decodable representation of the board in a player-relative (MINE/YOURS) frame (Li et al., 2023; Nanda et al., 2023). We ask whether this emergent linear world model survives a change of training objective: from next-token prediction to masked diffusion (Sahoo et al., 2024). On an identical GPT-2 backbone and identical data, we train an AR control and a masked diffusion language model (MDLM), reproduce the Li–Nanda linear/relative dissociation on the control, and probe the MDLM with the same battery. We find that the linear player-relative world model transfers to masked diffusion (the dissociation reproduces) but is substantially weaker (~0.79 vs. the AR’s 0.96 two-way accuracy). A Nanda-style causal intervention shows the representation is only minimally used: a statistically real but tiny and inconsistent effect (mean +0.02, 24% win-rate) against the AR’s strong, consistent +0.21 / 84%. Order-artifact controls confirm the representation is genuine rather than an artifact of an easy unmasking order. As a reproducibility contribution, this is a generalizability stress test of the Li–Nanda finding under a change of training objective, anchored throughout by random-weight and shuffled-label control tasks and by component ablations. We report a second implementation, in Rust on an independent interpretability stack, that reproduces our headline numbers. Code, data recipes, and model checkpoints are available anonymously for review at https://anonymous.4open.science/r/othello-mdlm-anon-DBC2/.