<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sultan-2023-knowledge">
    <titleInfo>
        <title>Knowledge Distillation \approx Label Smoothing: Fact or Fallacy?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Md</namePart>
        <namePart type="family">Sultan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Originally proposed as a method for knowledge transfer from one model to another, some recent studies have suggested that knowledge distillation (KD) is in fact a form of regularization. Perhaps the strongest argument of all for this new perspective comes from its apparent similarities with label smoothing (LS). Here we re-examine this stated equivalence between the two methods by comparing the predictive confidences of the models they train. Experiments on four text classification tasks involving models of different sizes show that: (a) In most settings, KD and LS drive model confidence in completely opposite directions, and (b) In KD, the student inherits not only its knowledge but also its confidence from the teacher, reinforcing the classical knowledge transfer view.</abstract>
    <identifier type="citekey">sultan-2023-knowledge</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.271</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.271/</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>4469</start>
            <end>4477</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Knowledge Distillation \approx Label Smoothing: Fact or Fallacy?
%A Sultan, Md
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F sultan-2023-knowledge
%X Originally proposed as a method for knowledge transfer from one model to another, some recent studies have suggested that knowledge distillation (KD) is in fact a form of regularization. Perhaps the strongest argument of all for this new perspective comes from its apparent similarities with label smoothing (LS). Here we re-examine this stated equivalence between the two methods by comparing the predictive confidences of the models they train. Experiments on four text classification tasks involving models of different sizes show that: (a) In most settings, KD and LS drive model confidence in completely opposite directions, and (b) In KD, the student inherits not only its knowledge but also its confidence from the teacher, reinforcing the classical knowledge transfer view.
%R 10.18653/v1/2023.emnlp-main.271
%U https://aclanthology.org/2023.emnlp-main.271/
%U https://doi.org/10.18653/v1/2023.emnlp-main.271
%P 4469-4477

Markdown (Informal)

[Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?](https://aclanthology.org/2023.emnlp-main.271/) (Sultan, EMNLP 2023)

• Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy? (Sultan, EMNLP 2023)

ACL

• Md Sultan. 2023. Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4469–4477, Singapore. Association for Computational Linguistics.