<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ahn-etal-2022-knowledge">
    <titleInfo>
        <title>Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jaimeen</namePart>
        <namePart type="family">Ahn</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hwaran</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jinhwa</namePart>
        <namePart type="family">Kim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alice</namePart>
        <namePart type="family">Oh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Christian</namePart>
            <namePart type="family">Hardmeier</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christine</namePart>
            <namePart type="family">Basta</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marta</namePart>
            <namePart type="given">R</namePart>
            <namePart type="family">Costa-jussà</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gabriel</namePart>
            <namePart type="family">Stanovsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hila</namePart>
            <namePart type="family">Gonen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Seattle, Washington</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Knowledge distillation is widely used to transfer the language understanding of a large model to a smaller model. However, after knowledge distillation, it was found that the smaller model is more biased by gender compared to the source large model. This paper studies what causes gender bias to increase after the knowledge distillation process. Moreover, we suggest applying a variant of the mixup on knowledge distillation, which is used to increase generalizability during the distillation process, not for augmentation. By doing so, we can significantly reduce the gender bias amplification after knowledge distillation. We also conduct an experiment on the GLUE benchmark to demonstrate that even if the mixup is applied, it does not have a significant adverse effect on the model’s performance.</abstract>
    <identifier type="citekey">ahn-etal-2022-knowledge</identifier>
    <identifier type="doi">10.18653/v1/2022.gebnlp-1.27</identifier>
    <location>
        <url>https://aclanthology.org/2022.gebnlp-1.27</url>
    </location>
    <part>
        <date>2022-07</date>
        <extent unit="page">
            <start>266</start>
            <end>272</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT
%A Ahn, Jaimeen
%A Lee, Hwaran
%A Kim, Jinhwa
%A Oh, Alice
%Y Hardmeier, Christian
%Y Basta, Christine
%Y Costa-jussà, Marta R.
%Y Stanovsky, Gabriel
%Y Gonen, Hila
%S Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)
%D 2022
%8 July
%I Association for Computational Linguistics
%C Seattle, Washington
%F ahn-etal-2022-knowledge
%X Knowledge distillation is widely used to transfer the language understanding of a large model to a smaller model. However, after knowledge distillation, it was found that the smaller model is more biased by gender compared to the source large model. This paper studies what causes gender bias to increase after the knowledge distillation process. Moreover, we suggest applying a variant of the mixup on knowledge distillation, which is used to increase generalizability during the distillation process, not for augmentation. By doing so, we can significantly reduce the gender bias amplification after knowledge distillation. We also conduct an experiment on the GLUE benchmark to demonstrate that even if the mixup is applied, it does not have a significant adverse effect on the model’s performance.
%R 10.18653/v1/2022.gebnlp-1.27
%U https://aclanthology.org/2022.gebnlp-1.27
%U https://doi.org/10.18653/v1/2022.gebnlp-1.27
%P 266-272

Markdown (Informal)

[Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT](https://aclanthology.org/2022.gebnlp-1.27) (Ahn et al., GeBNLP 2022)

• Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT (Ahn et al., GeBNLP 2022)

ACL

• Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. 2022. Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 266–272, Seattle, Washington. Association for Computational Linguistics.