<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="liu-zeldes-2023-gumsum">
    <titleInfo>
        <title>GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yang</namePart>
        <namePart type="given">Janet</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Amir</namePart>
        <namePart type="family">Zeldes</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Automatic summarization with pre-trained language models has led to impressively fluent results, but is prone to ‘hallucinations’, low performance on non-news genres, and outputs which are not exactly summaries. Targeting ACL 2023’s ‘Reality Check’ theme, we present GUMSum, a small but carefully crafted dataset of English summaries in 12 written and spoken genres for evaluation of abstractive summarization. Summaries are highly constrained, focusing on substitutive potential, factuality, and faithfulness. We present guidelines and evaluate human agreement as well as subjective judgments on recent system outputs, comparing general-domain untuned approaches, a fine-tuned one, and a prompt-based approach, to human performance. Results show that while GPT3 achieves impressive scores, it still underperforms humans, with varying quality across genres. Human judgments reveal different types of errors in supervised, prompted, and human-generated summaries, shedding light on the challenges of producing a good summary.</abstract>
    <identifier type="citekey">liu-zeldes-2023-gumsum</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-acl.593</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-acl.593</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>9315</start>
            <end>9327</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization
%A Liu, Yang Janet
%A Zeldes, Amir
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Findings of the Association for Computational Linguistics: ACL 2023
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F liu-zeldes-2023-gumsum
%X Automatic summarization with pre-trained language models has led to impressively fluent results, but is prone to ‘hallucinations’, low performance on non-news genres, and outputs which are not exactly summaries. Targeting ACL 2023’s ‘Reality Check’ theme, we present GUMSum, a small but carefully crafted dataset of English summaries in 12 written and spoken genres for evaluation of abstractive summarization. Summaries are highly constrained, focusing on substitutive potential, factuality, and faithfulness. We present guidelines and evaluate human agreement as well as subjective judgments on recent system outputs, comparing general-domain untuned approaches, a fine-tuned one, and a prompt-based approach, to human performance. Results show that while GPT3 achieves impressive scores, it still underperforms humans, with varying quality across genres. Human judgments reveal different types of errors in supervised, prompted, and human-generated summaries, shedding light on the challenges of producing a good summary.
%R 10.18653/v1/2023.findings-acl.593
%U https://aclanthology.org/2023.findings-acl.593
%U https://doi.org/10.18653/v1/2023.findings-acl.593
%P 9315-9327

Markdown (Informal)

[GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization](https://aclanthology.org/2023.findings-acl.593) (Liu & Zeldes, Findings 2023)

• GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization (Liu & Zeldes, Findings 2023)

ACL

• Yang Janet Liu and Amir Zeldes. 2023. GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9315–9327, Toronto, Canada. Association for Computational Linguistics.