<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="touileb-etal-2023-measuring">
    <titleInfo>
        <title>Measuring Normative and Descriptive Biases in Language Models Using Census Data</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Samia</namePart>
        <namePart type="family">Touileb</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lilja</namePart>
        <namePart type="family">Øvrelid</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Erik</namePart>
        <namePart type="family">Velldal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Andreas</namePart>
            <namePart type="family">Vlachos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isabelle</namePart>
            <namePart type="family">Augenstein</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dubrovnik, Croatia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models. Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment of reality. In this paper, we introduce an approach for measuring to what degree pre-trained language models are aligned to normative and descriptive occupational distributions. To this end, we use official demographic information about gender–occupation distributions provided by the national statistics agencies of France, Norway, United Kingdom, and the United States. We manually generate template-based sentences combining gendered pronouns and nouns with occupations, and subsequently probe a selection of ten language models covering the English, French, and Norwegian languages. The scoring system we introduce in this work is language independent, and can be used on any combination of template-based sentences, occupations, and languages. The approach could also be extended to other dimensions of national census data and other demographic variables.</abstract>
    <identifier type="citekey">touileb-etal-2023-measuring</identifier>
    <identifier type="doi">10.18653/v1/2023.eacl-main.164</identifier>
    <location>
        <url>https://aclanthology.org/2023.eacl-main.164</url>
    </location>
    <part>
        <date>2023-05</date>
        <extent unit="page">
            <start>2242</start>
            <end>2248</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Measuring Normative and Descriptive Biases in Language Models Using Census Data
%A Touileb, Samia
%A Øvrelid, Lilja
%A Velldal, Erik
%Y Vlachos, Andreas
%Y Augenstein, Isabelle
%S Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
%D 2023
%8 May
%I Association for Computational Linguistics
%C Dubrovnik, Croatia
%F touileb-etal-2023-measuring
%X We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models. Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment of reality. In this paper, we introduce an approach for measuring to what degree pre-trained language models are aligned to normative and descriptive occupational distributions. To this end, we use official demographic information about gender–occupation distributions provided by the national statistics agencies of France, Norway, United Kingdom, and the United States. We manually generate template-based sentences combining gendered pronouns and nouns with occupations, and subsequently probe a selection of ten language models covering the English, French, and Norwegian languages. The scoring system we introduce in this work is language independent, and can be used on any combination of template-based sentences, occupations, and languages. The approach could also be extended to other dimensions of national census data and other demographic variables.
%R 10.18653/v1/2023.eacl-main.164
%U https://aclanthology.org/2023.eacl-main.164
%U https://doi.org/10.18653/v1/2023.eacl-main.164
%P 2242-2248

Markdown (Informal)

[Measuring Normative and Descriptive Biases in Language Models Using Census Data](https://aclanthology.org/2023.eacl-main.164) (Touileb et al., EACL 2023)

• Measuring Normative and Descriptive Biases in Language Models Using Census Data (Touileb et al., EACL 2023)

ACL

• Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2023. Measuring Normative and Descriptive Biases in Language Models Using Census Data. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2242–2248, Dubrovnik, Croatia. Association for Computational Linguistics.