<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lu-etal-2021-rw-kd">
    <titleInfo>
        <title>RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Peng</namePart>
        <namePart type="family">Lu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Abbas</namePart>
        <namePart type="family">Ghaddar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ahmad</namePart>
        <namePart type="family">Rashid</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mehdi</namePart>
        <namePart type="family">Rezagholizadeh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ali</namePart>
        <namePart type="family">Ghodsi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Philippe</namePart>
        <namePart type="family">Langlais</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2021</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Marie-Francine</namePart>
            <namePart type="family">Moens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xuanjing</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucia</namePart>
            <namePart type="family">Specia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Scott</namePart>
            <namePart type="given">Wen-tau</namePart>
            <namePart type="family">Yih</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific fine-tuning phases of large neural language models. A student model is trained to minimize a convex combination of the prediction loss over the labels and another over the teacher output. However, most existing works either fix the interpolating weight between the two losses apriori or vary the weight using heuristics. In this work, we propose a novel sample-wise loss weighting method, RW-KD. A meta-learner, simultaneously trained with the student, adaptively re-weights the two losses for each sample. We demonstrate, on 7 datasets of the GLUE benchmark, that RW-KD outperforms other loss re-weighting methods for KD.</abstract>
    <identifier type="citekey">lu-etal-2021-rw-kd</identifier>
    <identifier type="doi">10.18653/v1/2021.findings-emnlp.270</identifier>
    <location>
        <url>https://aclanthology.org/2021.findings-emnlp.270</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>3145</start>
            <end>3152</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation
%A Lu, Peng
%A Ghaddar, Abbas
%A Rashid, Ahmad
%A Rezagholizadeh, Mehdi
%A Ghodsi, Ali
%A Langlais, Philippe
%Y Moens, Marie-Francine
%Y Huang, Xuanjing
%Y Specia, Lucia
%Y Yih, Scott Wen-tau
%S Findings of the Association for Computational Linguistics: EMNLP 2021
%D 2021
%8 November
%I Association for Computational Linguistics
%C Punta Cana, Dominican Republic
%F lu-etal-2021-rw-kd
%X Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific fine-tuning phases of large neural language models. A student model is trained to minimize a convex combination of the prediction loss over the labels and another over the teacher output. However, most existing works either fix the interpolating weight between the two losses apriori or vary the weight using heuristics. In this work, we propose a novel sample-wise loss weighting method, RW-KD. A meta-learner, simultaneously trained with the student, adaptively re-weights the two losses for each sample. We demonstrate, on 7 datasets of the GLUE benchmark, that RW-KD outperforms other loss re-weighting methods for KD.
%R 10.18653/v1/2021.findings-emnlp.270
%U https://aclanthology.org/2021.findings-emnlp.270
%U https://doi.org/10.18653/v1/2021.findings-emnlp.270
%P 3145-3152

Markdown (Informal)

[RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation](https://aclanthology.org/2021.findings-emnlp.270) (Lu et al., Findings 2021)

• RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation (Lu et al., Findings 2021)

ACL

• Peng Lu, Abbas Ghaddar, Ahmad Rashid, Mehdi Rezagholizadeh, Ali Ghodsi, and Philippe Langlais. 2021. RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3145–3152, Punta Cana, Dominican Republic. Association for Computational Linguistics.