<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="huidrom-etal-2024-differences-semantic">
    <titleInfo>
        <title>Differences in Semantic Errors Made by Different Types of Data-to-text Systems</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Rudali</namePart>
        <namePart type="family">Huidrom</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anya</namePart>
        <namePart type="family">Belz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Michela</namePart>
        <namePart type="family">Lorandi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 17th International Natural Language Generation Conference</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Saad</namePart>
            <namePart type="family">Mahamood</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nguyen</namePart>
            <namePart type="given">Le</namePart>
            <namePart type="family">Minh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Daphne</namePart>
            <namePart type="family">Ippolito</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Tokyo, Japan</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper, we investigate how different semantic, or content-related, errors made by different types of data-to-text systems differ in terms of number and type. In total, we examine 15 systems: three rule-based and 12 neural systems including two large language models without training or fine-tuning. All systems were tested on the English WebNLG dataset version 3.0. We use a semantic error taxonomy and the brat annotation tool to obtain word-span error annotations on a sample of system outputs. The annotations enable us to establish how many semantic errors different (types of) systems make and what specific types of errors they make, and thus to get an overall understanding of semantic strengths and weaknesses among various types of NLG systems. Among our main findings, we observe that symbolic (rule and template-based) systems make fewer semantic errors overall, non-LLM neural systems have better fluency and data coverage, but make more semantic errors, while LLM-based systems require improvement particularly in addressing superfluous.</abstract>
    <identifier type="citekey">huidrom-etal-2024-differences-semantic</identifier>
    <location>
        <url>https://aclanthology.org/2024.inlg-main.47</url>
    </location>
    <part>
        <date>2024-09</date>
        <extent unit="page">
            <start>609</start>
            <end>621</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Differences in Semantic Errors Made by Different Types of Data-to-text Systems
%A Huidrom, Rudali
%A Belz, Anya
%A Lorandi, Michela
%Y Mahamood, Saad
%Y Minh, Nguyen Le
%Y Ippolito, Daphne
%S Proceedings of the 17th International Natural Language Generation Conference
%D 2024
%8 September
%I Association for Computational Linguistics
%C Tokyo, Japan
%F huidrom-etal-2024-differences-semantic
%X In this paper, we investigate how different semantic, or content-related, errors made by different types of data-to-text systems differ in terms of number and type. In total, we examine 15 systems: three rule-based and 12 neural systems including two large language models without training or fine-tuning. All systems were tested on the English WebNLG dataset version 3.0. We use a semantic error taxonomy and the brat annotation tool to obtain word-span error annotations on a sample of system outputs. The annotations enable us to establish how many semantic errors different (types of) systems make and what specific types of errors they make, and thus to get an overall understanding of semantic strengths and weaknesses among various types of NLG systems. Among our main findings, we observe that symbolic (rule and template-based) systems make fewer semantic errors overall, non-LLM neural systems have better fluency and data coverage, but make more semantic errors, while LLM-based systems require improvement particularly in addressing superfluous.
%U https://aclanthology.org/2024.inlg-main.47
%P 609-621

Markdown (Informal)

[Differences in Semantic Errors Made by Different Types of Data-to-text Systems](https://aclanthology.org/2024.inlg-main.47) (Huidrom et al., INLG 2024)

• Differences in Semantic Errors Made by Different Types of Data-to-text Systems (Huidrom et al., INLG 2024)

ACL

• Rudali Huidrom, Anya Belz, and Michela Lorandi. 2024. Differences in Semantic Errors Made by Different Types of Data-to-text Systems. In Proceedings of the 17th International Natural Language Generation Conference, pages 609–621, Tokyo, Japan. Association for Computational Linguistics.