Valentin Noël
Author directory2026
Tokenization Sets the Measurement Scale for Cross-Lingual Diagnostics
Arij Riabi | Valentin Noël
The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)
Arij Riabi | Valentin Noël
The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)
Multilingual evaluation treats translated passages as the same input in different languages, and increasingly reads an internal diagnostic off the model while doing so. Tokenization can make that assumption false before the model answers, because the same passage occupies several times as many token positions in one language as in another, so a fixed window gives the model different amounts of text and gives the attention graph behind the diagnostic a different number of nodes. We derive a comparability criterion from the descriptors’ own definitions: on a parallel corpus the tokenizer sets the graph order, so a descriptor whose attainable range grows with graph order can separate languages while measuring segmentation rather than model computation, and because those ranges are analytic the criterion can be checked before any measurement is taken. Replaying a standard analysis against the audited one changes the answer, since most of its significant associations do not survive and about one in five reverse sign. The failures concentrate in the descriptors the criterion flags in advance rather than in spectral descriptors as such, and the same split appears among interpretability descriptors that use no graph at all. Applied to high frequency energy over 60 Belebele languages and three tokenizer matched Llama models, the audit leaves the resource associated profile varying across depth and withdraws the stronger readings the raw profiles invite, among them a stable peak layer, a replicated profile shape, a stable mid-depth effect and a causal mechanism. The practical consequence is that a cross-lingual comparison should establish that its measurements are in comparable units before it reports a difference.