Maria Matveev
Author directory2026
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
Felicia Körner | Maria Matveev | Florian Eichin | Gitta Kutyniok | Barbara Plank | Michael A. Hedderich
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Felicia Körner | Maria Matveev | Florian Eichin | Gitta Kutyniok | Barbara Plank | Michael A. Hedderich
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large language models exhibit impressive cross-lingual capabilities. However, *how* cross-lingual generalization emerges has only been analyzed through isolated factors and at sparse points during training, limiting our understanding — particularly of the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We use word-level translation as a testbed, introducing a novel dataset to trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which generalizing translation mechanisms develop and copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.