<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bavaresco-etal-2024-dont">
    <titleInfo>
        <title>Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Anna</namePart>
        <namePart type="family">Bavaresco</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alberto</namePart>
        <namePart type="family">Testoni</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Raquel</namePart>
        <namePart type="family">Fernández</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Image-based advertisements are complex multimodal stimuli that often contain unusual visual elements and figurative language. Previous research on automatic ad understanding has reported impressive zero-shot accuracy of contrastive vision-and-language models (VLMs) on an ad-explanation retrieval task. Here, we examine the original task setup and show that contrastive VLMs can solve it by exploiting grounding heuristics. To control for this confound, we introduce TRADE, a new evaluation test set with adversarial grounded explanations. While these explanations look implausible to humans, we show that they “fool” four different contrastive VLMs. Our findings highlight the need for an improved operationalisation of automatic ad understanding that truly evaluates VLMs’ multimodal reasoning abilities. We make our code and TRADE available at https://github.com/dmg-illc/trade.</abstract>
    <identifier type="citekey">bavaresco-etal-2024-dont</identifier>
    <location>
        <url>https://aclanthology.org/2024.acl-short.77</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>870</start>
            <end>879</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models
%A Bavaresco, Anna
%A Testoni, Alberto
%A Fernández, Raquel
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F bavaresco-etal-2024-dont
%X Image-based advertisements are complex multimodal stimuli that often contain unusual visual elements and figurative language. Previous research on automatic ad understanding has reported impressive zero-shot accuracy of contrastive vision-and-language models (VLMs) on an ad-explanation retrieval task. Here, we examine the original task setup and show that contrastive VLMs can solve it by exploiting grounding heuristics. To control for this confound, we introduce TRADE, a new evaluation test set with adversarial grounded explanations. While these explanations look implausible to humans, we show that they “fool” four different contrastive VLMs. Our findings highlight the need for an improved operationalisation of automatic ad understanding that truly evaluates VLMs’ multimodal reasoning abilities. We make our code and TRADE available at https://github.com/dmg-illc/trade.
%U https://aclanthology.org/2024.acl-short.77
%P 870-879

Markdown (Informal)

[Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models](https://aclanthology.org/2024.acl-short.77) (Bavaresco et al., ACL 2024)

• Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models (Bavaresco et al., ACL 2024)

ACL

• Anna Bavaresco, Alberto Testoni, and Raquel Fernández. 2024. Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 870–879, Bangkok, Thailand. Association for Computational Linguistics.