<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhou-etal-2023-predictive">
    <titleInfo>
        <title>A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yi</namePart>
        <namePart type="family">Zhou</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jose</namePart>
        <namePart type="family">Camacho-Collados</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Danushka</namePart>
        <namePart type="family">Bollegala</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work. However, multiple underlying factors are associated with an MLM such as its model size, size of the training data, training objectives, the domain from which pretraining data is sampled, tokenization, and languages present in the pretrained corpora, to name a few. It remains unclear as to which of those factors influence social biases that are learned by MLMs. To study the relationship between model factors and the social biases learned by an MLM, as well as the downstream task performance of the model, we conduct a comprehensive study over 39 pretrained MLMs covering different model sizes, training objectives, tokenization methods, training data domains and languages. Our results shed light on important factors often neglected in prior literature, such as tokenization or model objectives.</abstract>
    <identifier type="citekey">zhou-etal-2023-predictive</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.683</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.683</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>11082</start>
            <end>11100</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models
%A Zhou, Yi
%A Camacho-Collados, Jose
%A Bollegala, Danushka
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F zhou-etal-2023-predictive
%X Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work. However, multiple underlying factors are associated with an MLM such as its model size, size of the training data, training objectives, the domain from which pretraining data is sampled, tokenization, and languages present in the pretrained corpora, to name a few. It remains unclear as to which of those factors influence social biases that are learned by MLMs. To study the relationship between model factors and the social biases learned by an MLM, as well as the downstream task performance of the model, we conduct a comprehensive study over 39 pretrained MLMs covering different model sizes, training objectives, tokenization methods, training data domains and languages. Our results shed light on important factors often neglected in prior literature, such as tokenization or model objectives.
%R 10.18653/v1/2023.emnlp-main.683
%U https://aclanthology.org/2023.emnlp-main.683
%U https://doi.org/10.18653/v1/2023.emnlp-main.683
%P 11082-11100

Markdown (Informal)

[A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models](https://aclanthology.org/2023.emnlp-main.683) (Zhou et al., EMNLP 2023)

• A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models (Zhou et al., EMNLP 2023)

ACL

• Yi Zhou, Jose Camacho-Collados, and Danushka Bollegala. 2023. A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11082–11100, Singapore. Association for Computational Linguistics.