Ahmad Al Rabghi
Author directory2026
SAFIR: A Sub-Dialectal Benchmark for Saudi Arabic–English Machine Translation
Nour Qandos | Omer Nacar | Aya Hamod | Samar Ahmad | Rahaf Saeed Al Hallay | Ahmad Al Rabghi
Proceedings of The Fourth Arabic Natural Language Processing Conference
Nour Qandos | Omer Nacar | Aya Hamod | Samar Ahmad | Rahaf Saeed Al Hallay | Ahmad Al Rabghi
Proceedings of The Fourth Arabic Natural Language Processing Conference
Arabic machine translation benchmarks provide limited coverage of variation within Saudi Arabic. We introduce SAFIR, a Saudi-centric benchmark of 610 unique sentence pairs organized into 12 creator specific evaluation splits across five regional groups: Najdi, Hijazi, Southern, Northern, and Eastern. Native speakers authored the Arabic sentences and their English translations; 235 pairs received independent double review for dialect authenticity, translation adequacy, and fluency. We evaluate twelve systems, comprising losed-weight LLMs, open-weight LLMs, and dedicated MT models, in the zero-shot Saudi→English setting using reference-based metrics and GPT-4o judgments. Closed-weight LLMs obtain the strongest aggregate results. Gemini 2.5 Pro achieves the highest average BLEU (34.41), while GPT-5 achieves the highest BERTScore (83.60). Rankings vary across splits, and Southern varieties present particular challenges in judged adequacy. Differences between reference overlap, adequacy, and fluency support reporting performance by sampled variety and evaluation dimension. SAFIR and its leaderboard support more detailed assessment of Saudi Arabic translation.