Oleg Kachan

Author directory

2026

Multilingual machine translation models must adapt to the grammatical structure of the target language, yet how structural information shapes decoder computation remains unclear. Using the eigenvalue spectra of per-(layer, head) attention Gram matrices instead of raw attention weights, we show that cross-attention geometry in NLLB-200 is organized around the target language: these spectral features carry a strong, distributed signal about target-language word order (SVO vs. SOV) when English is the source, with near-perfect decodability across cross-validation. The signal is not concentrated in a small subset of heads, but is broadly distributed across layers and heads—yet it degrades sharply at every level of aggregation when English is the target. This asymmetry suggests that cross-attention geometry reflects the demands of target-side generation rather than source-side structure. Leave-one-language-out evaluation confirms that the signal generalizes to unseen languages, ruling out language identity as a confound.