Fatma Ezzahra Jlali
Author directory2026
Is TunisianMMLU Actually Tunisian? A Native-Speaker Audit of a Machine-Translated Dialect Benchmark
Fatma Ezzahra Jlali | Marek Šuppa
Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026)
Fatma Ezzahra Jlali | Marek Šuppa
Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026)
TunisianMMLU, the only MMLU-style benchmark for Tunisian Arabic (Derja), was machine-translated with no native-speaker validation. Scanning all 21,500 items with native-validated patterns, we flag 63.4% of the corpus; the largest signal, in 55.1% of items, is the Moroccan present-tense prefix, a form Tunisian does not have. The dominant defect is not translation noise but a different dialect. We then present the first native-speaker audit: 400 items stratified over all 44 subjects, labelled independently by two native speakers on six criteria. The non-author annotator marks 64.5% defective, both agree on 62.0%, and author adjudication yields 86.0%. Nearly all defects originate in translation, not in the source benchmarks. Six models corroborate the labels, scoring highest on items marked clean and lowest on unrepairable ones. Corrected text raises every above-chance model by 2.3 to 4.9 points (one gain survives multiple-test correction), leaves a chance-level Derja model unchanged, and narrows the gap between a dialect-adapted model and its base without changing rank order. We release TunisianMMLU-Verified: a corrected subset, corpus-wide flags, per-item labels, the annotation guideline, and all evaluation code.