<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sathe-etal-2024-unified">
    <titleInfo>
        <title>A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ashutosh</namePart>
        <namePart type="family">Sathe</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Prachi</namePart>
        <namePart type="family">Jain</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sunayana</namePart>
        <namePart type="family">Sitaram</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Vision-language models (VLMs) have gained widespread adoption in both industry and academia. In this study, we propose a unified framework for systematically evaluating gender, race, and age biases in VLMs with respect to professions. Our evaluation encompasses all supported inference modes of the recent VLMs, including image-to-text, text-to-text, text-to-image, and image-to-image. We create a synthetic, high-quality dataset comprising text and images that intentionally obscure gender, race, and age distinctions across various professions. The dataset includes action-based descriptions of each profession and serves as a benchmark for evaluating societal biases in vision-language models (VLMs). In our benchmarking of popular vision-language models (VLMs), we observe that different input-output modalities result in distinct bias magnitudes and directions. We hope our work will help guide future progress in improving VLMs to learn socially unbiased representations. We will release our data and code.</abstract>
    <identifier type="citekey">sathe-etal-2024-unified</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-emnlp.66</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>1208</start>
            <end>1249</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
%A Sathe, Ashutosh
%A Jain, Prachi
%A Sitaram, Sunayana
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Findings of the Association for Computational Linguistics: EMNLP 2024
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F sathe-etal-2024-unified
%X Vision-language models (VLMs) have gained widespread adoption in both industry and academia. In this study, we propose a unified framework for systematically evaluating gender, race, and age biases in VLMs with respect to professions. Our evaluation encompasses all supported inference modes of the recent VLMs, including image-to-text, text-to-text, text-to-image, and image-to-image. We create a synthetic, high-quality dataset comprising text and images that intentionally obscure gender, race, and age distinctions across various professions. The dataset includes action-based descriptions of each profession and serves as a benchmark for evaluating societal biases in vision-language models (VLMs). In our benchmarking of popular vision-language models (VLMs), we observe that different input-output modalities result in distinct bias magnitudes and directions. We hope our work will help guide future progress in improving VLMs to learn socially unbiased representations. We will release our data and code.
%U https://aclanthology.org/2024.findings-emnlp.66
%P 1208-1249

Markdown (Informal)

[A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models](https://aclanthology.org/2024.findings-emnlp.66) (Sathe et al., Findings 2024)

• A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models (Sathe et al., Findings 2024)

ACL

• Ashutosh Sathe, Prachi Jain, and Sunayana Sitaram. 2024. A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1208–1249, Miami, Florida, USA. Association for Computational Linguistics.