<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hassine-etal-2025-tounsibench">
    <titleInfo>
        <title>TounsiBench: Benchmarking Large Language Models for Tunisian Arabic</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Souha</namePart>
        <namePart type="given">Ben</namePart>
        <namePart type="family">Hassine</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Asma</namePart>
        <namePart type="family">Arrak</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marouene</namePart>
        <namePart type="family">Addhoum</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Steven</namePart>
        <namePart type="given">R</namePart>
        <namePart type="family">Wilson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Christos</namePart>
            <namePart type="family">Christodoulopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tanmoy</namePart>
            <namePart type="family">Chakraborty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Carolyn</namePart>
            <namePart type="family">Rose</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Violet</namePart>
            <namePart type="family">Peng</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Suzhou, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-332-6</identifier>
    </relatedItem>
    <abstract>In this work, we introduce the first benchmark for evaluating the capabilities of large language models (LLMs) in understanding and generating responses in Tunisian Arabic. To achieve this, we construct a dataset of Tunisian Arabic instructions and prompt ten widely-used LLMs that claim to support Arabic. We then assess the LLM responses through both human and LLM-based evaluations across four criteria: quality, correctness, relevance, and dialectal adherence. We analyze the agreement and correlation between these judgments and identify GPT-4o as our automated judge model based on its high correlation with human ratings, and generate a final leaderboard using this model. Our error analysis reveals that most LLMs struggle with recognizing and properly responding in Tunisian Arabic. To facilitate further research, we release our dataset, along with gold-standard human-written responses for all 744 instructions, and our evaluation framework, allowing others to benchmark their own models.</abstract>
    <identifier type="citekey">hassine-etal-2025-tounsibench</identifier>
    <location>
        <url>https://aclanthology.org/2025.emnlp-main.1756/</url>
    </location>
    <part>
        <date>2025-11</date>
        <extent unit="page">
            <start>34615</start>
            <end>34630</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T TounsiBench: Benchmarking Large Language Models for Tunisian Arabic
%A Hassine, Souha Ben
%A Arrak, Asma
%A Addhoum, Marouene
%A Wilson, Steven R.
%Y Christodoulopoulos, Christos
%Y Chakraborty, Tanmoy
%Y Rose, Carolyn
%Y Peng, Violet
%S Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
%D 2025
%8 November
%I Association for Computational Linguistics
%C Suzhou, China
%@ 979-8-89176-332-6
%F hassine-etal-2025-tounsibench
%X In this work, we introduce the first benchmark for evaluating the capabilities of large language models (LLMs) in understanding and generating responses in Tunisian Arabic. To achieve this, we construct a dataset of Tunisian Arabic instructions and prompt ten widely-used LLMs that claim to support Arabic. We then assess the LLM responses through both human and LLM-based evaluations across four criteria: quality, correctness, relevance, and dialectal adherence. We analyze the agreement and correlation between these judgments and identify GPT-4o as our automated judge model based on its high correlation with human ratings, and generate a final leaderboard using this model. Our error analysis reveals that most LLMs struggle with recognizing and properly responding in Tunisian Arabic. To facilitate further research, we release our dataset, along with gold-standard human-written responses for all 744 instructions, and our evaluation framework, allowing others to benchmark their own models.
%U https://aclanthology.org/2025.emnlp-main.1756/
%P 34615-34630

Markdown (Informal)

[TounsiBench: Benchmarking Large Language Models for Tunisian Arabic](https://aclanthology.org/2025.emnlp-main.1756/) (Hassine et al., EMNLP 2025)

• TounsiBench: Benchmarking Large Language Models for Tunisian Arabic (Hassine et al., EMNLP 2025)

ACL

• Souha Ben Hassine, Asma Arrak, Marouene Addhoum, and Steven R Wilson. 2025. TounsiBench: Benchmarking Large Language Models for Tunisian Arabic. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34615–34630, Suzhou, China. Association for Computational Linguistics.