<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="iter-etal-2020-pretraining">
    <titleInfo>
        <title>Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Dan</namePart>
        <namePart type="family">Iter</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kelvin</namePart>
        <namePart type="family">Guu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Larry</namePart>
        <namePart type="family">Lansing</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dan</namePart>
        <namePart type="family">Jurafsky</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent models for unsupervised representation learning of text have employed a number of techniques to improve contextual word representations but have put little focus on discourse-level representations. We propose Conpono, an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. Given an anchor sentence, our model is trained to predict the text k sentences away using a sampled-softmax objective where the candidates consist of neighboring sentences and sentences randomly sampled from the corpus. On the discourse representation benchmark DiscoEval, our model improves over the previous state-of-the-art by up to 13% and on average 4% absolute across 7 tasks. Our model is the same size as BERT-Base, but outperforms the much larger BERT-Large model and other more recent approaches that incorporate discourse. We also show that Conpono yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse: textual entailment (RTE), common sense reasoning (COPA) and reading comprehension (ReCoRD).</abstract>
    <identifier type="citekey">iter-etal-2020-pretraining</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.439</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.439</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>4859</start>
            <end>4870</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
%A Iter, Dan
%A Guu, Kelvin
%A Lansing, Larry
%A Jurafsky, Dan
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F iter-etal-2020-pretraining
%X Recent models for unsupervised representation learning of text have employed a number of techniques to improve contextual word representations but have put little focus on discourse-level representations. We propose Conpono, an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. Given an anchor sentence, our model is trained to predict the text k sentences away using a sampled-softmax objective where the candidates consist of neighboring sentences and sentences randomly sampled from the corpus. On the discourse representation benchmark DiscoEval, our model improves over the previous state-of-the-art by up to 13% and on average 4% absolute across 7 tasks. Our model is the same size as BERT-Base, but outperforms the much larger BERT-Large model and other more recent approaches that incorporate discourse. We also show that Conpono yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse: textual entailment (RTE), common sense reasoning (COPA) and reading comprehension (ReCoRD).
%R 10.18653/v1/2020.acl-main.439
%U https://aclanthology.org/2020.acl-main.439
%U https://doi.org/10.18653/v1/2020.acl-main.439
%P 4859-4870

Markdown (Informal)

[Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://aclanthology.org/2020.acl-main.439) (Iter et al., ACL 2020)

• Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models (Iter et al., ACL 2020)

ACL

• Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. 2020. Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4859–4870, Online. Association for Computational Linguistics.