<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="marafatto-navigli-2026-mind">
    <titleInfo>
        <title>Mind the Language Gap: Assessing LLM Safety in Italian</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Elena</namePart>
        <namePart type="family">Marafatto</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Roberto</namePart>
        <namePart type="family">Navigli</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fifteenth Language Resources and Evaluation Conference</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Stelios</namePart>
            <namePart type="family">Piperidis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Núria</namePart>
            <namePart type="family">Bel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Henk</namePart>
            <namePart type="family">van den Heuvel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nancy</namePart>
            <namePart type="family">Ide</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Simon</namePart>
            <namePart type="family">Krek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonio</namePart>
            <namePart type="family">Toral</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resource Association</publisher>
            <place>
                <placeTerm type="text">Palma de Mallorca, Spain</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The rapid diffusion of Large Language Models (LLMs) across linguistic and cultural contexts underscores the need for systematic safety evaluations beyond English. As LLMs are increasingly applied in multilingual settings, ensuring their safe and appropriate behavior in other languages is essential. This paper presents a methodology for building safety evaluation datasets that comprehensively cover the full spectrum of sensitive topics relevant to LLM safety. The resulting resources include a collection of Italian Wikipedia pages encompassing all major categories of sensitive content, and a companion dataset containing three challenging Italian-language questions per page designed to probe model behavior on high-risk issues. Each prompt was annotated into four safety outcome categories: correct refusal, safe informative, unsafe, and ambiguous. Together, these datasets provide a robust foundation for evaluating and benchmarking LLM safety in Italian. To demonstrate their utility, we used them to assess four LLMs, identifying systematic differences in refusal consistency and compliance across sensitive domains. To support transparency and reproducibility, we release a public repository containing the list of categorized Italian Wikipedia pages, the automatically generated prompts, and the standard prompt template used for safety testing. With this work, we aim to advance language-specific safety assessment and support the responsible, culturally grounded deployment of LLMs beyond English.</abstract>
    <identifier type="citekey">marafatto-navigli-2026-mind</identifier>
    <identifier type="doi">10.63317/4mvfemxb3wvv</identifier>
    <location>
        <url>https://aclanthology.org/2026.lrec-1.365/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>4647</start>
            <end>4657</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Mind the Language Gap: Assessing LLM Safety in Italian
%A Marafatto, Elena
%A Navigli, Roberto
%Y Piperidis, Stelios
%Y Bel, Núria
%Y van den Heuvel, Henk
%Y Ide, Nancy
%Y Krek, Simon
%Y Toral, Antonio
%S Proceedings of the Fifteenth Language Resources and Evaluation Conference
%D 2026
%8 May
%I ELRA Language Resource Association
%C Palma de Mallorca, Spain
%F marafatto-navigli-2026-mind
%X The rapid diffusion of Large Language Models (LLMs) across linguistic and cultural contexts underscores the need for systematic safety evaluations beyond English. As LLMs are increasingly applied in multilingual settings, ensuring their safe and appropriate behavior in other languages is essential. This paper presents a methodology for building safety evaluation datasets that comprehensively cover the full spectrum of sensitive topics relevant to LLM safety. The resulting resources include a collection of Italian Wikipedia pages encompassing all major categories of sensitive content, and a companion dataset containing three challenging Italian-language questions per page designed to probe model behavior on high-risk issues. Each prompt was annotated into four safety outcome categories: correct refusal, safe informative, unsafe, and ambiguous. Together, these datasets provide a robust foundation for evaluating and benchmarking LLM safety in Italian. To demonstrate their utility, we used them to assess four LLMs, identifying systematic differences in refusal consistency and compliance across sensitive domains. To support transparency and reproducibility, we release a public repository containing the list of categorized Italian Wikipedia pages, the automatically generated prompts, and the standard prompt template used for safety testing. With this work, we aim to advance language-specific safety assessment and support the responsible, culturally grounded deployment of LLMs beyond English.
%R 10.63317/4mvfemxb3wvv
%U https://aclanthology.org/2026.lrec-1.365/
%U https://doi.org/10.63317/4mvfemxb3wvv
%P 4647-4657

Markdown (Informal)

[Mind the Language Gap: Assessing LLM Safety in Italian](https://aclanthology.org/2026.lrec-1.365/) (Marafatto & Navigli, LREC 2026)

• Mind the Language Gap: Assessing LLM Safety in Italian (Marafatto & Navigli, LREC 2026)

ACL

• Elena Marafatto and Roberto Navigli. 2026. Mind the Language Gap: Assessing LLM Safety in Italian. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 4647–4657, Palma de Mallorca, Spain. ELRA Language Resource Association.