<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="schmidtova-etal-2026-hotelcheckspan">
    <titleInfo>
        <title>HotelCheckSpan: A Benchmark Dataset for LLM Faithfulness</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Patricia</namePart>
        <namePart type="family">Schmidtova</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ondrej</namePart>
        <namePart type="family">Dusek</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Saad</namePart>
        <namePart type="family">Mahamood</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fifteenth Language Resources and Evaluation Conference</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Stelios</namePart>
            <namePart type="family">Piperidis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Núria</namePart>
            <namePart type="family">Bel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Henk</namePart>
            <namePart type="family">van den Heuvel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nancy</namePart>
            <namePart type="family">Ide</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Simon</namePart>
            <namePart type="family">Krek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonio</namePart>
            <namePart type="family">Toral</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resource Association</publisher>
            <place>
                <placeTerm type="text">Palma de Mallorca, Spain</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Hallucinations are among the most persistent and challenging issues in large language model (LLM) outputs. This particularly holds in domains that combine both objective and subjective content, such as hotel descriptions, that are intended to be enticing advertisements for the hotel. Distinguishing between factual errors and interpretative exaggeration is often subtle, complicating both human and automated evaluation. To address this, we present HotelCheckSpan, the first span-level faithfulness dataset for the hotel domain. Each example aggregates one or more hotel descriptions, and human-annotated summaries are labeled with three error types: Incorrect, Misleading, and Not Checkable. By marking the precise spans where errors occur, the dataset captures fine-grained information about the nature of hallucinations and factual inconsistencies. In addition to human annotations, we collect span-level judgments from multiple LLMs, enabling direct human–model comparisons. Our analysis shows that inter-annotator agreement varies substantially across aggregation levels: example-level agreement can mask subtle span-level disagreements, while soft and hard F1 variants highlight discrepancies in both span placement and error categorization. HotelCheckSpan provides a benchmark for studying ambiguity and disagreement, validating automatic faithfulness metrics, and evaluating LLMs as judges, offering a rich resource for research on faithfulness, subjectivity, and annotation practices in mixed-content domains</abstract>
    <identifier type="citekey">schmidtova-etal-2026-hotelcheckspan</identifier>
    <identifier type="doi">10.63317/3x4p5r52zmvw</identifier>
    <location>
        <url>https://aclanthology.org/2026.lrec-1.782/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>9973</start>
            <end>9987</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T HotelCheckSpan: A Benchmark Dataset for LLM Faithfulness
%A Schmidtova, Patricia
%A Dusek, Ondrej
%A Mahamood, Saad
%Y Piperidis, Stelios
%Y Bel, Núria
%Y van den Heuvel, Henk
%Y Ide, Nancy
%Y Krek, Simon
%Y Toral, Antonio
%S Proceedings of the Fifteenth Language Resources and Evaluation Conference
%D 2026
%8 May
%I ELRA Language Resource Association
%C Palma de Mallorca, Spain
%F schmidtova-etal-2026-hotelcheckspan
%X Hallucinations are among the most persistent and challenging issues in large language model (LLM) outputs. This particularly holds in domains that combine both objective and subjective content, such as hotel descriptions, that are intended to be enticing advertisements for the hotel. Distinguishing between factual errors and interpretative exaggeration is often subtle, complicating both human and automated evaluation. To address this, we present HotelCheckSpan, the first span-level faithfulness dataset for the hotel domain. Each example aggregates one or more hotel descriptions, and human-annotated summaries are labeled with three error types: Incorrect, Misleading, and Not Checkable. By marking the precise spans where errors occur, the dataset captures fine-grained information about the nature of hallucinations and factual inconsistencies. In addition to human annotations, we collect span-level judgments from multiple LLMs, enabling direct human–model comparisons. Our analysis shows that inter-annotator agreement varies substantially across aggregation levels: example-level agreement can mask subtle span-level disagreements, while soft and hard F1 variants highlight discrepancies in both span placement and error categorization. HotelCheckSpan provides a benchmark for studying ambiguity and disagreement, validating automatic faithfulness metrics, and evaluating LLMs as judges, offering a rich resource for research on faithfulness, subjectivity, and annotation practices in mixed-content domains
%R 10.63317/3x4p5r52zmvw
%U https://aclanthology.org/2026.lrec-1.782/
%U https://doi.org/10.63317/3x4p5r52zmvw
%P 9973-9987

Markdown (Informal)

[HotelCheckSpan: A Benchmark Dataset for LLM Faithfulness](https://aclanthology.org/2026.lrec-1.782/) (Schmidtova et al., LREC 2026)

• HotelCheckSpan: A Benchmark Dataset for LLM Faithfulness (Schmidtova et al., LREC 2026)

ACL

• Patricia Schmidtova, Ondrej Dusek, and Saad Mahamood. 2026. HotelCheckSpan: A Benchmark Dataset for LLM Faithfulness. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 9973–9987, Palma de Mallorca, Spain. ELRA Language Resource Association.