BALSAM: A Platform for Benchmarking Arabic Large Language Models

Rawan Nasser Almatham, Kareem Mohamed Darwish, Raghad Al-Rasheed, Waad Thuwaini Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, Norah A. Alzahrani, Eman Albilali, Nizar Habash, Abdelrahman Mustafa El-Sheikh, Muhammad Elmallah, Hamdy Mubarak, Zaid Alyafeai, Mohamed Anwar, Haonan Li, Ahmed Abdelali, Nora Altwairesh, Maram Hasanain, Abdulmohsen Al-Thubaity, Shady Shehata, Bashar Alhafni, Injy Hamed, Go Inoue, Khalid N. Elmadani, Ossama Obeid, Fatima Haouari, Tamer Elsayed, Emad A. Alghamdi, Khalid Almubarak, Saied Alshahrani, Ola Aljareh, Safa Alajlan, Areej Alshaqarawi, Maryam Alshihri, Sultana Alghurabi, Atikah Alzeghayer, Afrah Altamimi, Abdullah Alfaifi, Abdulrahman M Alosaimy

Abstract
The impressive advancement of Large Language Models (LLMs) in English has not been matched across all languages. In particular, LLM performance in Arabic lags behind, due to data scarcity, linguistic diversity of Arabic and its dialects, morphological complexity, etc. Progress is further hindered by the quality of Arabic benchmarks, which typically rely on static, publicly available data, lack comprehensive task coverage, or do not provide dedicated platforms with blind test sets. This makes it challenging to measure actual progress and to mitigate data contamination. Here, we aim to bridge these gaps. In particular, we introduce BALSAM, a comprehensive, community-driven benchmark aimed at advancing Arabic LLM development and evaluation. It includes 78 NLP tasks from 14 broad categories, with 52K examples divided into 37K test and 15K development, and a centralized, transparent platform for blind evaluation. We envision BALSAM as a unifying platform that sets standards and promotes collaborative research to advance Arabic LLM capabilities.
Anthology ID:
2025.arabicnlp-main.21
Volume:
Proceedings of The Third Arabic Natural Language Processing Conference
Month:
November
Year:
2025
Address:
Suzhou, China
Editors:
Kareem Darwish, Ahmed Ali, Ibrahim Abu Farha, Samia Touileb, Imed Zitouni, Ahmed Abdelali, Sharefah Al-Ghamdi, Sakhar Alkhereyf, Wajdi Zaghouani, Salam Khalifa, Badr AlKhamissi, Rawan Almatham, Injy Hamed, Zaid Alyafeai, Areeb Alowisheq, Go Inoue, Khalil Mrini, Waad Alshammari
Venue:
ArabicNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
258–277
Language:
URL:
https://aclanthology.org/2025.arabicnlp-main.21/
DOI:
10.18653/v1/2025.arabicnlp-main.21
Bibkey:
Cite (ACL):
Rawan Nasser Almatham, Kareem Mohamed Darwish, Raghad Al-Rasheed, Waad Thuwaini Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, Norah A. Alzahrani, Eman Albilali, Nizar Habash, Abdelrahman Mustafa El-Sheikh, Muhammad Elmallah, Hamdy Mubarak, Zaid Alyafeai, Mohamed Anwar, Haonan Li, Ahmed Abdelali, Nora Altwairesh, Maram Hasanain, Abdulmohsen Al-Thubaity, Shady Shehata, Bashar Alhafni, Injy Hamed, Go Inoue, Khalid N. Elmadani, Ossama Obeid, Fatima Haouari, Tamer Elsayed, Emad A. Alghamdi, Khalid Almubarak, Saied Alshahrani, Ola Aljareh, Safa Alajlan, Areej Alshaqarawi, Maryam Alshihri, Sultana Alghurabi, Atikah Alzeghayer, Afrah Altamimi, Abdullah Alfaifi, and Abdulrahman M Alosaimy. 2025. BALSAM: A Platform for Benchmarking Arabic Large Language Models. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 258–277, Suzhou, China. Association for Computational Linguistics.
Cite (Informal):
BALSAM: A Platform for Benchmarking Arabic Large Language Models (Almatham et al., ArabicNLP 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.arabicnlp-main.21.pdf
PDF Cite Search Fix data