<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="jeong-etal-2024-simple">
    <titleInfo>
        <title>Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yoo</namePart>
        <namePart type="given">Hyun</namePart>
        <namePart type="family">Jeong</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Myeong</namePart>
        <namePart type="given">Soo</namePart>
        <namePart type="family">Han</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dong-Kyu</namePart>
        <namePart type="family">Chae</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Matthew</namePart>
            <namePart type="family">Purver</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper, we proposes a simple, tricky method to improve sentence representation of unsupervised contrastive learning. Even though contrastive learning has achieved great performances in both visual representation learning (VRL) and sentence representation learning (SRL) fields, we focus on the fact that there is a gap between characteristics and training dynamics of VRL and SRL. We first examine the role of temperature to bridge the gap between VRL and SRL, and find some temperature-dependent elements in SRL; i.e., a higher temperature causes overfitting of the uniformity while improving the alignment in earlier phase of training. Then, we design a temperature cool-down technique based on this observation, which helps PLMs to be more suitable for contrastive learning via preparation of uniform representation space. Our experimental results on widely-utilized benchmarks demonstrate the effectiveness and extensiblity of our method.</abstract>
    <identifier type="citekey">jeong-etal-2024-simple</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-eacl.37</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>550</start>
            <end>559</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning
%A Jeong, Yoo Hyun
%A Han, Myeong Soo
%A Chae, Dong-Kyu
%Y Graham, Yvette
%Y Purver, Matthew
%S Findings of the Association for Computational Linguistics: EACL 2024
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F jeong-etal-2024-simple
%X In this paper, we proposes a simple, tricky method to improve sentence representation of unsupervised contrastive learning. Even though contrastive learning has achieved great performances in both visual representation learning (VRL) and sentence representation learning (SRL) fields, we focus on the fact that there is a gap between characteristics and training dynamics of VRL and SRL. We first examine the role of temperature to bridge the gap between VRL and SRL, and find some temperature-dependent elements in SRL; i.e., a higher temperature causes overfitting of the uniformity while improving the alignment in earlier phase of training. Then, we design a temperature cool-down technique based on this observation, which helps PLMs to be more suitable for contrastive learning via preparation of uniform representation space. Our experimental results on widely-utilized benchmarks demonstrate the effectiveness and extensiblity of our method.
%U https://aclanthology.org/2024.findings-eacl.37
%P 550-559

Markdown (Informal)

[Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning](https://aclanthology.org/2024.findings-eacl.37) (Jeong et al., Findings 2024)

• Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning (Jeong et al., Findings 2024)

ACL

• Yoo Hyun Jeong, Myeong Soo Han, and Dong-Kyu Chae. 2024. Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning. In Findings of the Association for Computational Linguistics: EACL 2024, pages 550–559, St. Julian’s, Malta. Association for Computational Linguistics.