<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bugliarello-etal-2023-weakly">
    <titleInfo>
        <title>Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Emanuele</namePart>
        <namePart type="family">Bugliarello</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Aida</namePart>
        <namePart type="family">Nematzadeh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lisa</namePart>
        <namePart type="family">Hendricks</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. In this work, we take a step further and explore how we can tap into supervision from small-scale visual relation data. In particular, we propose two pretraining approaches to contextualise visual entities in a multimodal setup. With verbalised scene graphs, we transform visual relation triplets into structured captions, and treat them as additional image descriptions. With masked relation prediction, we further encourage relating entities from image regions with visually masked contexts. When applied to strong baselines pretrained on large amounts of Web data, zero-shot evaluations on both coarse-grained and fine-grained tasks show the efficacy of our methods in learning multimodal representations from weakly-supervised relations data.</abstract>
    <identifier type="citekey">bugliarello-etal-2023-weakly</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.184</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.184</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>3052</start>
            <end>3071</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining
%A Bugliarello, Emanuele
%A Nematzadeh, Aida
%A Hendricks, Lisa
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F bugliarello-etal-2023-weakly
%X Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. In this work, we take a step further and explore how we can tap into supervision from small-scale visual relation data. In particular, we propose two pretraining approaches to contextualise visual entities in a multimodal setup. With verbalised scene graphs, we transform visual relation triplets into structured captions, and treat them as additional image descriptions. With masked relation prediction, we further encourage relating entities from image regions with visually masked contexts. When applied to strong baselines pretrained on large amounts of Web data, zero-shot evaluations on both coarse-grained and fine-grained tasks show the efficacy of our methods in learning multimodal representations from weakly-supervised relations data.
%R 10.18653/v1/2023.emnlp-main.184
%U https://aclanthology.org/2023.emnlp-main.184
%U https://doi.org/10.18653/v1/2023.emnlp-main.184
%P 3052-3071

Markdown (Informal)

[Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining](https://aclanthology.org/2023.emnlp-main.184) (Bugliarello et al., EMNLP 2023)

• Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining (Bugliarello et al., EMNLP 2023)

ACL

• Emanuele Bugliarello, Aida Nematzadeh, and Lisa Hendricks. 2023. Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3052–3071, Singapore. Association for Computational Linguistics.