<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kwako-etal-2022-using">
    <titleInfo>
        <title>Using Item Response Theory to Measure Gender and Racial Bias of a BERT-based Automated English Speech Assessment System</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Alexander</namePart>
        <namePart type="family">Kwako</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yixin</namePart>
        <namePart type="family">Wan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jieyu</namePart>
        <namePart type="family">Zhao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kai-Wei</namePart>
        <namePart type="family">Chang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Li</namePart>
        <namePart type="family">Cai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mark</namePart>
        <namePart type="family">Hansen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Seattle, Washington</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent advances in natural language processing and transformer-based models have made it easier to implement accurate, automated English speech assessments. Yet, without careful examination, applications of these models may exacerbate social prejudices based on gender and race. This study addresses the need to examine potential biases of transformer-based models in the context of automated English speech assessment. For this purpose, we developed a BERT-based automated speech assessment system and investigated gender and racial bias of examinees’ automated scores. Gender and racial bias was measured by examining differential item functioning (DIF) using an item response theory framework. Preliminary results, which focused on a single verbal-response item, showed no statistically significant DIF based on gender or race for automated scores.</abstract>
    <identifier type="citekey">kwako-etal-2022-using</identifier>
    <identifier type="doi">10.18653/v1/2022.bea-1.1</identifier>
    <location>
        <url>https://aclanthology.org/2022.bea-1.1</url>
    </location>
    <part>
        <date>2022-07</date>
        <extent unit="page">
            <start>1</start>
            <end>7</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Using Item Response Theory to Measure Gender and Racial Bias of a BERT-based Automated English Speech Assessment System
%A Kwako, Alexander
%A Wan, Yixin
%A Zhao, Jieyu
%A Chang, Kai-Wei
%A Cai, Li
%A Hansen, Mark
%S Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)
%D 2022
%8 July
%I Association for Computational Linguistics
%C Seattle, Washington
%F kwako-etal-2022-using
%X Recent advances in natural language processing and transformer-based models have made it easier to implement accurate, automated English speech assessments. Yet, without careful examination, applications of these models may exacerbate social prejudices based on gender and race. This study addresses the need to examine potential biases of transformer-based models in the context of automated English speech assessment. For this purpose, we developed a BERT-based automated speech assessment system and investigated gender and racial bias of examinees’ automated scores. Gender and racial bias was measured by examining differential item functioning (DIF) using an item response theory framework. Preliminary results, which focused on a single verbal-response item, showed no statistically significant DIF based on gender or race for automated scores.
%R 10.18653/v1/2022.bea-1.1
%U https://aclanthology.org/2022.bea-1.1
%U https://doi.org/10.18653/v1/2022.bea-1.1
%P 1-7

Markdown (Informal)

[Using Item Response Theory to Measure Gender and Racial Bias of a BERT-based Automated English Speech Assessment System](https://aclanthology.org/2022.bea-1.1) (Kwako et al., BEA 2022)

• Using Item Response Theory to Measure Gender and Racial Bias of a BERT-based Automated English Speech Assessment System (Kwako et al., BEA 2022)

ACL

• Alexander Kwako, Yixin Wan, Jieyu Zhao, Kai-Wei Chang, Li Cai, and Mark Hansen. 2022. Using Item Response Theory to Measure Gender and Racial Bias of a BERT-based Automated English Speech Assessment System. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 1–7, Seattle, Washington. Association for Computational Linguistics.