LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking

Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly, Nadir Durrani, Firoj Alam


Abstract
The recent development and success of Large Language Models (LLMs) necessitate an evaluation of their performance across diverse NLP tasks in different languages. Although several frameworks have been developed and made publicly available, their customization capabilities for specific tasks and datasets are often complex for different users. In this study, we introduce the LLMeBench framework, which can be seamlessly customized to evaluate LLMs for any NLP task, regardless of language. The framework features generic dataset loaders, several model providers, and pre-implements most standard evaluation metrics. It supports in-context learning with zero- and few-shot settings. A specific dataset and task can be evaluated for a given LLM in less than 20 lines of code while allowing full flexibility to extend the framework for custom datasets, models, or tasks. The framework has been tested on 31 unique NLP tasks using 53 publicly available datasets within 90 experimental setups, involving approximately 296K data points. We open-sourced LLMeBench for the community (https://github.com/qcri/LLMeBench/) and a video demonstrating the framework is available online (https://youtu.be/9cC2m_abk3A).
Anthology ID:
2024.eacl-demo.23
Volume:
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations
Month:
March
Year:
2024
Address:
St. Julians, Malta
Editors:
Nikolaos Aletras, Orphee De Clercq
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
214–222
Language:
URL:
https://aclanthology.org/2024.eacl-demo.23
DOI:
Bibkey:
Cite (ACL):
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly, Nadir Durrani, and Firoj Alam. 2024. LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 214–222, St. Julians, Malta. Association for Computational Linguistics.
Cite (Informal):
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking (Dalvi et al., EACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.eacl-demo.23.pdf
Video:
 https://aclanthology.org/2024.eacl-demo.23.mp4