<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ilichev-etal-2021-multiple">
    <titleInfo>
        <title>Multiple Teacher Distillation for Robust and Greener Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Artur</namePart>
        <namePart type="family">Ilichev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nikita</namePart>
        <namePart type="family">Sorokin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Irina</namePart>
        <namePart type="family">Piontkovskaya</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Valentin</namePart>
        <namePart type="family">Malykh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ruslan</namePart>
            <namePart type="family">Mitkov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Galia</namePart>
            <namePart type="family">Angelova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>INCOMA Ltd.</publisher>
            <place>
                <placeTerm type="text">Held Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The language models nowadays are in the center of natural language processing progress. These models are mostly of significant size. There are successful attempts to reduce them, but at least some of these attempts rely on randomness. We propose a novel distillation procedure leveraging on multiple teachers usage which alleviates random seed dependency and makes the models more robust. We show that this procedure applied to TinyBERT and DistilBERT models improves their worst case results up to 2% while keeping almost the same best-case ones. The latter fact keeps true with a constraint on computational time, which is important to lessen the carbon footprint. In addition, we present the results of an application of the proposed procedure to a computer vision model ResNet, which shows that the statement keeps true in this totally different domain.</abstract>
    <identifier type="citekey">ilichev-etal-2021-multiple</identifier>
    <location>
        <url>https://aclanthology.org/2021.ranlp-1.68</url>
    </location>
    <part>
        <date>2021-09</date>
        <extent unit="page">
            <start>601</start>
            <end>610</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Multiple Teacher Distillation for Robust and Greener Models
%A Ilichev, Artur
%A Sorokin, Nikita
%A Piontkovskaya, Irina
%A Malykh, Valentin
%Y Mitkov, Ruslan
%Y Angelova, Galia
%S Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
%D 2021
%8 September
%I INCOMA Ltd.
%C Held Online
%F ilichev-etal-2021-multiple
%X The language models nowadays are in the center of natural language processing progress. These models are mostly of significant size. There are successful attempts to reduce them, but at least some of these attempts rely on randomness. We propose a novel distillation procedure leveraging on multiple teachers usage which alleviates random seed dependency and makes the models more robust. We show that this procedure applied to TinyBERT and DistilBERT models improves their worst case results up to 2% while keeping almost the same best-case ones. The latter fact keeps true with a constraint on computational time, which is important to lessen the carbon footprint. In addition, we present the results of an application of the proposed procedure to a computer vision model ResNet, which shows that the statement keeps true in this totally different domain.
%U https://aclanthology.org/2021.ranlp-1.68
%P 601-610

Markdown (Informal)

[Multiple Teacher Distillation for Robust and Greener Models](https://aclanthology.org/2021.ranlp-1.68) (Ilichev et al., RANLP 2021)

• Multiple Teacher Distillation for Robust and Greener Models (Ilichev et al., RANLP 2021)

ACL

• Artur Ilichev, Nikita Sorokin, Irina Piontkovskaya, and Valentin Malykh. 2021. Multiple Teacher Distillation for Robust and Greener Models. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 601–610, Held Online. INCOMA Ltd..