<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yang-ma-2022-improving">
    <titleInfo>
        <title>Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Chenghao</namePart>
        <namePart type="family">Yang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xuezhe</namePart>
        <namePart type="family">Ma</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yoav</namePart>
            <namePart type="family">Goldberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yue</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Fine-tuning over large pretrained language models (PLMs) has established many state-of-the-art results. Despite its superior performance, such fine-tuning can be unstable, resulting in significant variance in performance and potential risks for practical applications. Previous works have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs, which indicates iteratively fine-tuning layers in a top-down manner is a promising solution. In this paper, we first point out that this method does not always work out due to the different convergence speeds of different layers/modules. Inspired by this observation, we propose a simple component-wise gradient norm clipping method to adjust the convergence speed for different components. Experiment results demonstrate that our method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability. The codebase can be found at https://github.com/yangalan123/FineTuningStability.</abstract>
    <identifier type="citekey">yang-ma-2022-improving</identifier>
    <identifier type="doi">10.18653/v1/2022.emnlp-main.322</identifier>
    <location>
        <url>https://aclanthology.org/2022.emnlp-main.322</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>4854</start>
            <end>4859</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping
%A Yang, Chenghao
%A Ma, Xuezhe
%Y Goldberg, Yoav
%Y Kozareva, Zornitsa
%Y Zhang, Yue
%S Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates
%F yang-ma-2022-improving
%X Fine-tuning over large pretrained language models (PLMs) has established many state-of-the-art results. Despite its superior performance, such fine-tuning can be unstable, resulting in significant variance in performance and potential risks for practical applications. Previous works have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs, which indicates iteratively fine-tuning layers in a top-down manner is a promising solution. In this paper, we first point out that this method does not always work out due to the different convergence speeds of different layers/modules. Inspired by this observation, we propose a simple component-wise gradient norm clipping method to adjust the convergence speed for different components. Experiment results demonstrate that our method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability. The codebase can be found at https://github.com/yangalan123/FineTuningStability.
%R 10.18653/v1/2022.emnlp-main.322
%U https://aclanthology.org/2022.emnlp-main.322
%U https://doi.org/10.18653/v1/2022.emnlp-main.322
%P 4854-4859

Markdown (Informal)

[Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping](https://aclanthology.org/2022.emnlp-main.322) (Yang & Ma, EMNLP 2022)

• Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping (Yang & Ma, EMNLP 2022)

ACL

• Chenghao Yang and Xuezhe Ma. 2022. Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4854–4859, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.