<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="krishna-etal-2021-pretraining-summarization">
    <titleInfo>
        <title>Does Pretraining for Summarization Require Knowledge Transfer?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Kundan</namePart>
        <namePart type="family">Krishna</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jeffrey</namePart>
        <namePart type="family">Bigham</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zachary</namePart>
        <namePart type="given">C</namePart>
        <namePart type="family">Lipton</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2021</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Marie-Francine</namePart>
            <namePart type="family">Moens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xuanjing</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucia</namePart>
            <namePart type="family">Specia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Scott</namePart>
            <namePart type="given">Wen-tau</namePart>
            <namePart type="family">Yih</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pretraining techniques leveraging enormous datasets have driven recent advances in text summarization. While folk explanations suggest that knowledge transfer accounts for pretraining’s benefits, little is known about why it works or what makes a pretraining task or dataset suitable. In this paper, we challenge the knowledge transfer story, showing that pretraining on documents consisting of character n-grams selected at random, we can nearly match the performance of models pretrained on real corpora. This work holds the promise of eliminating upstream corpora, which may alleviate some concerns over offensive language, bias, and copyright issues. To see whether the small residual benefit of using real data could be accounted for by the structure of the pretraining task, we design several tasks motivated by a qualitative study of summarization corpora. However, these tasks confer no appreciable benefit, leaving open the possibility of a small role for knowledge transfer.</abstract>
    <identifier type="citekey">krishna-etal-2021-pretraining-summarization</identifier>
    <identifier type="doi">10.18653/v1/2021.findings-emnlp.273</identifier>
    <location>
        <url>https://aclanthology.org/2021.findings-emnlp.273</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>3178</start>
            <end>3189</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Does Pretraining for Summarization Require Knowledge Transfer?
%A Krishna, Kundan
%A Bigham, Jeffrey
%A Lipton, Zachary C.
%Y Moens, Marie-Francine
%Y Huang, Xuanjing
%Y Specia, Lucia
%Y Yih, Scott Wen-tau
%S Findings of the Association for Computational Linguistics: EMNLP 2021
%D 2021
%8 November
%I Association for Computational Linguistics
%C Punta Cana, Dominican Republic
%F krishna-etal-2021-pretraining-summarization
%X Pretraining techniques leveraging enormous datasets have driven recent advances in text summarization. While folk explanations suggest that knowledge transfer accounts for pretraining’s benefits, little is known about why it works or what makes a pretraining task or dataset suitable. In this paper, we challenge the knowledge transfer story, showing that pretraining on documents consisting of character n-grams selected at random, we can nearly match the performance of models pretrained on real corpora. This work holds the promise of eliminating upstream corpora, which may alleviate some concerns over offensive language, bias, and copyright issues. To see whether the small residual benefit of using real data could be accounted for by the structure of the pretraining task, we design several tasks motivated by a qualitative study of summarization corpora. However, these tasks confer no appreciable benefit, leaving open the possibility of a small role for knowledge transfer.
%R 10.18653/v1/2021.findings-emnlp.273
%U https://aclanthology.org/2021.findings-emnlp.273
%U https://doi.org/10.18653/v1/2021.findings-emnlp.273
%P 3178-3189

Markdown (Informal)

[Does Pretraining for Summarization Require Knowledge Transfer?](https://aclanthology.org/2021.findings-emnlp.273) (Krishna et al., Findings 2021)

• Does Pretraining for Summarization Require Knowledge Transfer? (Krishna et al., Findings 2021)

ACL

• Kundan Krishna, Jeffrey Bigham, and Zachary C. Lipton. 2021. Does Pretraining for Summarization Require Knowledge Transfer?. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3178–3189, Punta Cana, Dominican Republic. Association for Computational Linguistics.