Aman Mussa
Author directory2026
Same Problem, Different Answer: Accuracy Hides Cross-Lingual Inconsistency in Multilingual Inference
Ahsan Habib Shihab | Assel Ospan | Aman Mussa | Madina Mansurova
Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026)
Ahsan Habib Shihab | Assel Ospan | Aman Mussa | Madina Mansurova
Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026)
Multilingual natural language understanding is almost always reported as per-language accuracy: a model is judged reliable in a language if it scores well there. This lens overstates reliability, because it aggregates over items and hides item-level cross-lingual inconsistency: the same reasoning problem receiving different answers in different languages. On the parallel XNLI benchmark (identical premise–hypothesis pairs in 15 languages) we evaluate two off-the-shelf zero-shot multilingual NLI models from different architecture families. Despite per-language accuracies of 80.8% and 71.3%, the models return a unanimous answer across all 15 languages for only 38.9% and 21.9% of test problems; the prediction flips across languages for a majority of items (61.1%, 78.1%), and agreement beyond chance is only moderate (κ = 0.70, 0.55). These flips are typically silent errors: on most split items the cross-lingual majority is correct (80.2%, 76.0%), so a reader in the dissenting language receives a wrong answer the model gets right elsewhere, and they fall disproportionately on lower-resource languages (flip involvement vs. a resource proxy: r = −0.67, −0.69). Contrary to what the negation-is-hard literature would predict, this instability is not concentrated on explicit pragmatic cues: across both models, items containing negation or quantifiers flip slightly less than cue-free items, not more. Item-level cross-lingual consistency, not aggregate accuracy, is the reliability metric that matters when the same content must be understood identically across languages. Because the instability is pervasive rather than localized, it cannot be mitigated by flagging “hard” inputs.