<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="gaci-etal-2022-debiasing">
    <titleInfo>
        <title>Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yacine</namePart>
        <namePart type="family">Gaci</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Boualem</namePart>
        <namePart type="family">Benatallah</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Fabio</namePart>
        <namePart type="family">Casati</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Khalid</namePart>
        <namePart type="family">Benabdeslem</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yoav</namePart>
            <namePart type="family">Goldberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yue</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Natural Language Processing (NLP) models are found to exhibit discriminatory stereotypes across many social constructs, e.g. gender and race. In comparison to the progress made in reducing bias from static word embeddings, fairness in sentence-level text encoders received little consideration despite their wider applicability in contemporary NLP tasks. In this paper, we propose a debiasing method for pre-trained text encoders that both reduces social stereotypes, and inflicts next to no semantic damage. Unlike previous studies that directly manipulate the embeddings, we suggest to dive deeper into the operation of these encoders, and pay more attention to the way they pay attention to different social groups. We find that stereotypes are also encoded in the attention layer. Then, we work on model debiasing by redistributing the attention scores of a text encoder such that it forgets any preference to historically advantaged groups, and attends to all social classes with the same intensity. Our experiments confirm that reducing bias from attention effectively mitigates it from the model’s text representations.</abstract>
    <identifier type="citekey">gaci-etal-2022-debiasing</identifier>
    <identifier type="doi">10.18653/v1/2022.emnlp-main.651</identifier>
    <location>
        <url>https://aclanthology.org/2022.emnlp-main.651</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>9582</start>
            <end>9602</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention
%A Gaci, Yacine
%A Benatallah, Boualem
%A Casati, Fabio
%A Benabdeslem, Khalid
%Y Goldberg, Yoav
%Y Kozareva, Zornitsa
%Y Zhang, Yue
%S Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates
%F gaci-etal-2022-debiasing
%X Natural Language Processing (NLP) models are found to exhibit discriminatory stereotypes across many social constructs, e.g. gender and race. In comparison to the progress made in reducing bias from static word embeddings, fairness in sentence-level text encoders received little consideration despite their wider applicability in contemporary NLP tasks. In this paper, we propose a debiasing method for pre-trained text encoders that both reduces social stereotypes, and inflicts next to no semantic damage. Unlike previous studies that directly manipulate the embeddings, we suggest to dive deeper into the operation of these encoders, and pay more attention to the way they pay attention to different social groups. We find that stereotypes are also encoded in the attention layer. Then, we work on model debiasing by redistributing the attention scores of a text encoder such that it forgets any preference to historically advantaged groups, and attends to all social classes with the same intensity. Our experiments confirm that reducing bias from attention effectively mitigates it from the model’s text representations.
%R 10.18653/v1/2022.emnlp-main.651
%U https://aclanthology.org/2022.emnlp-main.651
%U https://doi.org/10.18653/v1/2022.emnlp-main.651
%P 9582-9602

Markdown (Informal)

[Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention](https://aclanthology.org/2022.emnlp-main.651) (Gaci et al., EMNLP 2022)

• Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention (Gaci et al., EMNLP 2022)

ACL

• Yacine Gaci, Boualem Benatallah, Fabio Casati, and Khalid Benabdeslem. 2022. Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9582–9602, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.