Current machine translation (MT) evaluation practices largely assume that high lexical and semantic fidelity implies preservation of meaning. We question this assumption by introducing a framework for detecting and quantifying translation-induced distortion—the systematic alteration of a text’s subjective properties during translation. Focusing on stance as a socially consequential property, we formalize stance preservation as an invariance problem and adapt two classical statistical tests, McNemar’s test and the two-proportion Z-test, to diagnose systematic opinion shifts between source texts and their translations. Unlike standard MT metrics such as BLEU or COMET, which prioritize surface similarity and adequacy, our approach explicitly targets preservation of subjective meaning. In controlled experiments with synthetically distorted translations, we demonstrate that the proposed tests are sensitive to graded levels of stance manipulation. We apply our framework to evaluate twelve multilingual models and find that none reliably preserve stance across all tested language directions. Our findings reveal a critical gap in current MT evaluation practices and highlight the need for explicit evaluation of subjective meaning preservation in socially and politically sensitive contexts.

We investigate how model scale shapes lexical acquisition in LLMs, introducing trajectory comparison as a methodological tool and three new convergence metrics. Tracking surprisal dynamics across four Pythia model sizes and 153 training checkpoints, we report two main findings. First, scale drives a global rise in inter-word surprisal coherence: synonym curves become more correlated at larger scales, but baseline correction reveals that the advantage of semantic relatedness over random pairs is invariant with scale, indicating that what scales is broad inter-word coherence rather than semantic abstraction specifically. Second, learning dynamics are shaped primarily by training data rather than by model capacity: surprisal trajectories are highly similar across model sizes, and our new metrics confirm that larger models converge faster but not along qualitatively different paths. Together, these results call for baseline-corrected analyses and longitudinal evaluation when assessing lexical competence in LLMs.

Un grand nombre de travaux s’appuient sur l’analyse des courbes de surprise pour évaluer la manière dont les modèles de langue capture le sens des mots au cours de leur apprentissage. Toutefois, cette approche ne considère que la capacité d’un modèle à prédire un mot dans des contextes appropriés, sans prendre en compte sa capacité à ne pas produire ce mot dans des contextes inappropriés. Pour combler cette lacune, nous introduisons une nouvelle mesure complémentaire, que nous appelons l’anti-surpris, qui évalue la capacité d’un modèle à ne pas utiliser un mot dans des contextes où il serait surprenant voire erroné. Nous montrons que l’analyse conjointe des courbes de surprise et d’anti-surprise permet de mieux caractériser l’acquisition du lexique par les modèles de langue.

Nous proposons une nouvelle méthode pour mesurer les biais politiques des grands modèles de langue multilingues pour la traduction automatique, l’aide à la rédaction et le résumé automatique. Nous nous appuyons sur une représentation dense des opinions politiques exprimées dans les textes, apprise de façon faiblement supervisée.

Many studies have explored when and how LLMs learn to use specific words, primarily by examining their learning curves. While these curves capture a model’s capacity to use words correctly in context, they often neglect the equally important skill of avoiding incorrect usage. In this paper, we introduce a new metric, anti-surprisal, which measures a model’s capacity to refrain from using words in inappropriate or unexpected contexts. By examining both correct usage and error avoidance, we offer a more comprehensive perspective on the learning dynamics of LLMs.