<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="akyurek-etal-2022-challenges">
    <titleInfo>
        <title>Challenges in Measuring Bias via Open-Ended Language Generation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Afra</namePart>
        <namePart type="given">Feyza</namePart>
        <namePart type="family">Akyürek</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Muhammed</namePart>
        <namePart type="given">Yusuf</namePart>
        <namePart type="family">Kocyigit</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sejin</namePart>
        <namePart type="family">Paik</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Derry</namePart>
        <namePart type="given">Tanti</namePart>
        <namePart type="family">Wijaya</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Seattle, Washington</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Researchers have devised numerous ways to quantify social biases vested in pretrained language models. As some language models are capable of generating coherent completions given a set of textual prompts, several prompting datasets have been proposed to measure biases between social groups—posing language generation as a way of identifying biases. In this opinion paper, we analyze how specific choices of prompt sets, metrics, automatic tools and sampling strategies affect bias results. We find out that the practice of measuring biases through text completion is prone to yielding contradicting results under different experiment settings. We additionally provide recommendations for reporting biases in open-ended language generation for a more complete outlook of biases exhibited by a given language model. Code to reproduce the results is released under https://github.com/feyzaakyurek/bias-textgen.</abstract>
    <identifier type="citekey">akyurek-etal-2022-challenges</identifier>
    <identifier type="doi">10.18653/v1/2022.gebnlp-1.9</identifier>
    <location>
        <url>https://aclanthology.org/2022.gebnlp-1.9</url>
    </location>
    <part>
        <date>2022-07</date>
        <extent unit="page">
            <start>76</start>
            <end>76</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Challenges in Measuring Bias via Open-Ended Language Generation
%A Akyürek, Afra Feyza
%A Kocyigit, Muhammed Yusuf
%A Paik, Sejin
%A Wijaya, Derry Tanti
%S Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)
%D 2022
%8 July
%I Association for Computational Linguistics
%C Seattle, Washington
%F akyurek-etal-2022-challenges
%X Researchers have devised numerous ways to quantify social biases vested in pretrained language models. As some language models are capable of generating coherent completions given a set of textual prompts, several prompting datasets have been proposed to measure biases between social groups—posing language generation as a way of identifying biases. In this opinion paper, we analyze how specific choices of prompt sets, metrics, automatic tools and sampling strategies affect bias results. We find out that the practice of measuring biases through text completion is prone to yielding contradicting results under different experiment settings. We additionally provide recommendations for reporting biases in open-ended language generation for a more complete outlook of biases exhibited by a given language model. Code to reproduce the results is released under https://github.com/feyzaakyurek/bias-textgen.
%R 10.18653/v1/2022.gebnlp-1.9
%U https://aclanthology.org/2022.gebnlp-1.9
%U https://doi.org/10.18653/v1/2022.gebnlp-1.9
%P 76-76

Markdown (Informal)

[Challenges in Measuring Bias via Open-Ended Language Generation](https://aclanthology.org/2022.gebnlp-1.9) (Akyürek et al., GeBNLP 2022)

• Challenges in Measuring Bias via Open-Ended Language Generation (Akyürek et al., GeBNLP 2022)

ACL

• Afra Feyza Akyürek, Muhammed Yusuf Kocyigit, Sejin Paik, and Derry Tanti Wijaya. 2022. Challenges in Measuring Bias via Open-Ended Language Generation. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 76–76, Seattle, Washington. Association for Computational Linguistics.