<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="fan-etal-2024-biasalert">
    <titleInfo>
        <title>BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Zhiting</namePart>
        <namePart type="family">Fan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ruizhe</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ruiling</namePart>
        <namePart type="family">Xu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zuozhu</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Evaluating the bias of LLMs becomes more crucial with their rapid development. However, existing evaluation approaches rely on fixed-form outputs and cannot adapt to the flexible open-text generation scenarios of LLMs (e.g., sentence completion and question answering). To address this, we introduce BiasAlert, a plug-and-play tool designed to detect social bias in open-text generations of LLMs. BiasAlert integrates external human knowledge with its inherent reasoning capabilities to detect bias reliably. Extensive experiments demonstrate that BiasAlert significantly outperforms existing state-of-the-art methods like GPT-4-as-Judge in detecting bias. Furthermore, through application studies, we showcase the utility of BiasAlert in reliable LLM fairness evaluation and bias mitigation across various scenarios. Model and code will be publicly released.</abstract>
    <identifier type="citekey">fan-etal-2024-biasalert</identifier>
    <identifier type="doi">10.18653/v1/2024.emnlp-main.820</identifier>
    <location>
        <url>https://aclanthology.org/2024.emnlp-main.820/</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>14778</start>
            <end>14790</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs
%A Fan, Zhiting
%A Chen, Ruizhe
%A Xu, Ruiling
%A Liu, Zuozhu
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F fan-etal-2024-biasalert
%X Evaluating the bias of LLMs becomes more crucial with their rapid development. However, existing evaluation approaches rely on fixed-form outputs and cannot adapt to the flexible open-text generation scenarios of LLMs (e.g., sentence completion and question answering). To address this, we introduce BiasAlert, a plug-and-play tool designed to detect social bias in open-text generations of LLMs. BiasAlert integrates external human knowledge with its inherent reasoning capabilities to detect bias reliably. Extensive experiments demonstrate that BiasAlert significantly outperforms existing state-of-the-art methods like GPT-4-as-Judge in detecting bias. Furthermore, through application studies, we showcase the utility of BiasAlert in reliable LLM fairness evaluation and bias mitigation across various scenarios. Model and code will be publicly released.
%R 10.18653/v1/2024.emnlp-main.820
%U https://aclanthology.org/2024.emnlp-main.820/
%U https://doi.org/10.18653/v1/2024.emnlp-main.820
%P 14778-14790

Markdown (Informal)

[BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs](https://aclanthology.org/2024.emnlp-main.820/) (Fan et al., EMNLP 2024)

• BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs (Fan et al., EMNLP 2024)

ACL

• Zhiting Fan, Ruizhe Chen, Ruiling Xu, and Zuozhu Liu. 2024. BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14778–14790, Miami, Florida, USA. Association for Computational Linguistics.