<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zakizadeh-etal-2023-difair">
    <titleInfo>
        <title>DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mahdi</namePart>
        <namePart type="family">Zakizadeh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kaveh</namePart>
        <namePart type="family">Miandoab</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mohammad</namePart>
        <namePart type="family">Pilehvar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Numerous debiasing techniques have been proposed to mitigate the gender bias that is prevalent in pretrained language models. These are often evaluated on datasets that check the extent to which the model is gender-neutral in its predictions. Importantly, this evaluation protocol overlooks the possible adverse impact of bias mitigation on useful gender knowledge. To fill this gap, we propose **DiFair**, a manually curated dataset based on masked language modeling objectives. **DiFair** allows us to introduce a unified metric, *gender invariance score*, that not only quantifies a model’s biased behavior, but also checks if useful gender knowledge is preserved. We use **DiFair** as a benchmark for a number of widely-used pretained language models and debiasing techniques. Experimental results corroborate previous findings on the existing gender biases, while also demonstrating that although debiasing techniques ameliorate the issue of gender bias, this improvement usually comes at the price of lowering useful gender knowledge of the model.</abstract>
    <identifier type="citekey">zakizadeh-etal-2023-difair</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.127</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.127</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>1897</start>
            <end>1914</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias
%A Zakizadeh, Mahdi
%A Miandoab, Kaveh
%A Pilehvar, Mohammad
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F zakizadeh-etal-2023-difair
%X Numerous debiasing techniques have been proposed to mitigate the gender bias that is prevalent in pretrained language models. These are often evaluated on datasets that check the extent to which the model is gender-neutral in its predictions. Importantly, this evaluation protocol overlooks the possible adverse impact of bias mitigation on useful gender knowledge. To fill this gap, we propose **DiFair**, a manually curated dataset based on masked language modeling objectives. **DiFair** allows us to introduce a unified metric, *gender invariance score*, that not only quantifies a model’s biased behavior, but also checks if useful gender knowledge is preserved. We use **DiFair** as a benchmark for a number of widely-used pretained language models and debiasing techniques. Experimental results corroborate previous findings on the existing gender biases, while also demonstrating that although debiasing techniques ameliorate the issue of gender bias, this improvement usually comes at the price of lowering useful gender knowledge of the model.
%R 10.18653/v1/2023.findings-emnlp.127
%U https://aclanthology.org/2023.findings-emnlp.127
%U https://doi.org/10.18653/v1/2023.findings-emnlp.127
%P 1897-1914

Markdown (Informal)

[DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias](https://aclanthology.org/2023.findings-emnlp.127) (Zakizadeh et al., Findings 2023)

• DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias (Zakizadeh et al., Findings 2023)

ACL

• Mahdi Zakizadeh, Kaveh Miandoab, and Mohammad Pilehvar. 2023. DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1897–1914, Singapore. Association for Computational Linguistics.