<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhang-etal-2024-gaining">
    <titleInfo>
        <title>Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xiao</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Chunliu</namePart>
        <namePart type="family">Wang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rik</namePart>
        <namePart type="family">van Noord</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Johan</namePart>
        <namePart type="family">Bos</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Claire</namePart>
            <namePart type="family">Bonial</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Julia</namePart>
            <namePart type="family">Bonn</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jena</namePart>
            <namePart type="given">D</namePart>
            <namePart type="family">Hwang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA and ICCL</publisher>
            <place>
                <placeTerm type="text">Torino, Italia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The Parallel Meaning Bank (PMB) serves as a corpus for semantic processing with a focus on semantic parsing and text generation. Currently, we witness an excellent performance of neural parsers and generators on the PMB. This might suggest that such semantic processing tasks have by and large been solved. We argue that this is not the case and that performance scores from the past on the PMB are inflated by non-optimal data splits and test sets that are too easy. In response, we introduce several changes. First, instead of the prior random split, we propose a more systematic splitting approach to improve the reliability of the standard test data. Second, except for the standard test set, we also propose two challenge sets: one with longer texts including discourse structure, and one that addresses compositional generalization. We evaluate five neural models for semantic parsing and meaning-to-text generation. Our results show that model performance declines (in some cases dramatically) on the challenge sets, revealing the limitations of neural models when confronting such challenges.</abstract>
    <identifier type="citekey">zhang-etal-2024-gaining</identifier>
    <location>
        <url>https://aclanthology.org/2024.dmr-1.17</url>
    </location>
    <part>
        <date>2024-05</date>
        <extent unit="page">
            <start>162</start>
            <end>175</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks
%A Zhang, Xiao
%A Wang, Chunliu
%A van Noord, Rik
%A Bos, Johan
%Y Bonial, Claire
%Y Bonn, Julia
%Y Hwang, Jena D.
%S Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024
%D 2024
%8 May
%I ELRA and ICCL
%C Torino, Italia
%F zhang-etal-2024-gaining
%X The Parallel Meaning Bank (PMB) serves as a corpus for semantic processing with a focus on semantic parsing and text generation. Currently, we witness an excellent performance of neural parsers and generators on the PMB. This might suggest that such semantic processing tasks have by and large been solved. We argue that this is not the case and that performance scores from the past on the PMB are inflated by non-optimal data splits and test sets that are too easy. In response, we introduce several changes. First, instead of the prior random split, we propose a more systematic splitting approach to improve the reliability of the standard test data. Second, except for the standard test set, we also propose two challenge sets: one with longer texts including discourse structure, and one that addresses compositional generalization. We evaluate five neural models for semantic parsing and meaning-to-text generation. Our results show that model performance declines (in some cases dramatically) on the challenge sets, revealing the limitations of neural models when confronting such challenges.
%U https://aclanthology.org/2024.dmr-1.17
%P 162-175

Markdown (Informal)

[Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks](https://aclanthology.org/2024.dmr-1.17) (Zhang et al., DMR-WS 2024)

• Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks (Zhang et al., DMR-WS 2024)

ACL

• Xiao Zhang, Chunliu Wang, Rik van Noord, and Johan Bos. 2024. Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks. In Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024, pages 162–175, Torino, Italia. ELRA and ICCL.