<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="fang-xie-2022-end">
    <titleInfo>
        <title>An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hongchao</namePart>
        <namePart type="family">Fang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pengtao</namePart>
        <namePart type="family">Xie</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Self-supervised learning (SSL) methods such as Word2vec, BERT, and GPT have shown great effectiveness in language understanding. Contrastive learning, as a recent SSL approach, has attracted increasing attention in NLP. Contrastive learning learns data representations by predicting whether two augmented data instances are generated from the same original data example. Previous contrastive learning methods perform data augmentation and contrastive learning separately. As a result, the augmented data may not be optimal for contrastive learning. To address this problem, we propose a four-level optimization framework that performs data augmentation and contrastive learning end-to-end, to enable the augmented data to be tailored to the contrastive learning task. This framework consists of four learning stages, including training machine translation models for sentence augmentation, pretraining a text encoder using contrastive learning, finetuning a text classification model, and updating weights of translation data by minimizing the validation loss of the classification model, which are performed in a unified way. Experiments on datasets in the GLUE benchmark (Wang et al., 2018a) and on datasets used in Gururangan et al. (2020) demonstrate the effectiveness of our method.</abstract>
    <identifier type="citekey">fang-xie-2022-end</identifier>
    <identifier type="doi">10.1162/tacl_a_00521</identifier>
    <location>
        <url>https://aclanthology.org/2022.tacl-1.76</url>
    </location>
    <part>
        <date>2022</date>
        <detail type="volume"><number>10</number></detail>
        <extent unit="page">
            <start>1324</start>
            <end>1340</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding
%A Fang, Hongchao
%A Xie, Pengtao
%J Transactions of the Association for Computational Linguistics
%D 2022
%V 10
%I MIT Press
%C Cambridge, MA
%F fang-xie-2022-end
%X Self-supervised learning (SSL) methods such as Word2vec, BERT, and GPT have shown great effectiveness in language understanding. Contrastive learning, as a recent SSL approach, has attracted increasing attention in NLP. Contrastive learning learns data representations by predicting whether two augmented data instances are generated from the same original data example. Previous contrastive learning methods perform data augmentation and contrastive learning separately. As a result, the augmented data may not be optimal for contrastive learning. To address this problem, we propose a four-level optimization framework that performs data augmentation and contrastive learning end-to-end, to enable the augmented data to be tailored to the contrastive learning task. This framework consists of four learning stages, including training machine translation models for sentence augmentation, pretraining a text encoder using contrastive learning, finetuning a text classification model, and updating weights of translation data by minimizing the validation loss of the classification model, which are performed in a unified way. Experiments on datasets in the GLUE benchmark (Wang et al., 2018a) and on datasets used in Gururangan et al. (2020) demonstrate the effectiveness of our method.
%R 10.1162/tacl_a_00521
%U https://aclanthology.org/2022.tacl-1.76
%U https://doi.org/10.1162/tacl_a_00521
%P 1324-1340

Markdown (Informal)

[An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding](https://aclanthology.org/2022.tacl-1.76) (Fang & Xie, TACL 2022)

• An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (Fang & Xie, TACL 2022)

ACL

• Hongchao Fang and Pengtao Xie. 2022. An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding. Transactions of the Association for Computational Linguistics, 10:1324–1340.