<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhu-etal-2021-gaml">
    <titleInfo>
        <title>GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Wei</namePart>
        <namePart type="family">Zhu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xiaoling</namePart>
        <namePart type="family">Wang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yuan</namePart>
        <namePart type="family">Ni</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Guotong</namePart>
        <namePart type="family">Xie</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Marie-Francine</namePart>
            <namePart type="family">Moens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xuanjing</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucia</namePart>
            <namePart type="family">Specia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Scott</namePart>
            <namePart type="given">Wen-tau</namePart>
            <namePart type="family">Yih</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online and Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this work, we propose a novel framework, Gradient Aligned Mutual Learning BERT (GAML-BERT), for improving the early exiting of BERT. GAML-BERT’s contributions are two-fold. We conduct a set of pilot experiments, which shows that mutual knowledge distillation between a shallow exit and a deep exit leads to better performances for both. From this observation, we use mutual learning to improve BERT’s early exiting performances, that is, we ask each exit of a multi-exit BERT to distill knowledge from each other. Second, we propose GA, a novel training method that aligns the gradients from knowledge distillation to cross-entropy losses. Extensive experiments are conducted on the GLUE benchmark, which shows that our GAML-BERT can significantly outperform the state-of-the-art (SOTA) BERT early exiting methods.</abstract>
    <identifier type="citekey">zhu-etal-2021-gaml</identifier>
    <identifier type="doi">10.18653/v1/2021.emnlp-main.242</identifier>
    <location>
        <url>https://aclanthology.org/2021.emnlp-main.242</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>3033</start>
            <end>3044</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning
%A Zhu, Wei
%A Wang, Xiaoling
%A Ni, Yuan
%A Xie, Guotong
%Y Moens, Marie-Francine
%Y Huang, Xuanjing
%Y Specia, Lucia
%Y Yih, Scott Wen-tau
%S Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
%D 2021
%8 November
%I Association for Computational Linguistics
%C Online and Punta Cana, Dominican Republic
%F zhu-etal-2021-gaml
%X In this work, we propose a novel framework, Gradient Aligned Mutual Learning BERT (GAML-BERT), for improving the early exiting of BERT. GAML-BERT’s contributions are two-fold. We conduct a set of pilot experiments, which shows that mutual knowledge distillation between a shallow exit and a deep exit leads to better performances for both. From this observation, we use mutual learning to improve BERT’s early exiting performances, that is, we ask each exit of a multi-exit BERT to distill knowledge from each other. Second, we propose GA, a novel training method that aligns the gradients from knowledge distillation to cross-entropy losses. Extensive experiments are conducted on the GLUE benchmark, which shows that our GAML-BERT can significantly outperform the state-of-the-art (SOTA) BERT early exiting methods.
%R 10.18653/v1/2021.emnlp-main.242
%U https://aclanthology.org/2021.emnlp-main.242
%U https://doi.org/10.18653/v1/2021.emnlp-main.242
%P 3033-3044

Markdown (Informal)

[GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning](https://aclanthology.org/2021.emnlp-main.242) (Zhu et al., EMNLP 2021)

• GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning (Zhu et al., EMNLP 2021)

ACL

• Wei Zhu, Xiaoling Wang, Yuan Ni, and Guotong Xie. 2021. GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3033–3044, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.