Archit Rastogi
Author directory2026
A Reliability Audit of Aggregated MTEB Scores: Coverage Asymmetry and Language-Specific Rank Inversions Across 17 Languages
Archit Rastogi
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Archit Rastogi
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Practitioners select multilingual embedding models from aggregated MTEB leaderboard scores, but the same global score can correspond to very different per-language behaviours. We audit MTEB(Multilingual) retrieval rankings against language-specific rankings on 17 languages across three nested model rosters (restricted n=7; deployment-relevant curated n=25; broad MMTEB-eligible n=28-57), using Kendall 𝜏 and pairwise inversion rate. At deployment-realistic roster size, median pairwise inversion rate is 33% (per-language Kendall 𝜏 reliable after Benjamini-Hochberg FDR in 11/17 languages); the extended tier shows higher agreement (𝜏=0.69), reflecting its multilingual-first composition. Per-language task coverage is asymmetric, and inversion rate decreases with the number of MTEB retrieval tasks available for a language (Spearman 𝜌=-0.59 curated). A language-balanced aggregator (per-language mean first, then macro-average) drops the median inversion rate from 33% to 14.5% (leave-one-out, 𝜏=0.71), the lowest among the held-out aggregators we test and our most actionable mitigation. A cluster of hidden-failure models, headed by granite-311m (16/17 languages) and harrier-0.6b (10/17), ranks in the global top half but collapses to the local bottom quartile, with a distinctively shallow English-multilingual NDCG gap. We release the three-tier audit data, a multilingual RAG benchmark (IT/JA/HI × finance/law), and the analysis pipeline.