<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="rosch-libovicky-2022-probing">
    <titleInfo>
        <title>Probing the Role of Positional Information in Vision-Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Philipp</namePart>
        <namePart type="given">J</namePart>
        <namePart type="family">Rösch</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jindřich</namePart>
        <namePart type="family">Libovický</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: NAACL 2022</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Seattle, United States</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In most Vision-Language models (VL), the understanding of the image structure is enabled by injecting the position information (PI) about objects in the image. In our case study of LXMERT, a state-of-the-art VL model, we probe the use of the PI in the representation and study its effect on Visual Question Answering. We show that the model is not capable of leveraging the PI for the image-text matching task on a challenge set where only position differs. Yet, our experiments with probing confirm that the PI is indeed present in the representation. We introduce two strategies to tackle this: (i) Positional Information Pre-training and (ii) Contrastive Learning on PI using Cross-Modality Matching. Doing so, the model can correctly classify if images with detailed PI statements match. Additionally to the 2D information from bounding boxes, we introduce the object’s depth as new feature for a better object localization in the space. Even though we were able to improve the model properties as defined by our probes, it only has a negligible effect on the downstream performance. Our results thus highlight an important issue of multimodal modeling: the mere presence of information detectable by a probing classifier is not a guarantee that the information is available in a cross-modal setup.</abstract>
    <identifier type="citekey">rosch-libovicky-2022-probing</identifier>
    <identifier type="doi">10.18653/v1/2022.findings-naacl.77</identifier>
    <location>
        <url>https://aclanthology.org/2022.findings-naacl.77</url>
    </location>
    <part>
        <date>2022-07</date>
        <extent unit="page">
            <start>1031</start>
            <end>1041</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Probing the Role of Positional Information in Vision-Language Models
%A Rösch, Philipp J.
%A Libovický, Jindřich
%S Findings of the Association for Computational Linguistics: NAACL 2022
%D 2022
%8 July
%I Association for Computational Linguistics
%C Seattle, United States
%F rosch-libovicky-2022-probing
%X In most Vision-Language models (VL), the understanding of the image structure is enabled by injecting the position information (PI) about objects in the image. In our case study of LXMERT, a state-of-the-art VL model, we probe the use of the PI in the representation and study its effect on Visual Question Answering. We show that the model is not capable of leveraging the PI for the image-text matching task on a challenge set where only position differs. Yet, our experiments with probing confirm that the PI is indeed present in the representation. We introduce two strategies to tackle this: (i) Positional Information Pre-training and (ii) Contrastive Learning on PI using Cross-Modality Matching. Doing so, the model can correctly classify if images with detailed PI statements match. Additionally to the 2D information from bounding boxes, we introduce the object’s depth as new feature for a better object localization in the space. Even though we were able to improve the model properties as defined by our probes, it only has a negligible effect on the downstream performance. Our results thus highlight an important issue of multimodal modeling: the mere presence of information detectable by a probing classifier is not a guarantee that the information is available in a cross-modal setup.
%R 10.18653/v1/2022.findings-naacl.77
%U https://aclanthology.org/2022.findings-naacl.77
%U https://doi.org/10.18653/v1/2022.findings-naacl.77
%P 1031-1041

Markdown (Informal)

[Probing the Role of Positional Information in Vision-Language Models](https://aclanthology.org/2022.findings-naacl.77) (Rösch & Libovický, Findings 2022)

• Probing the Role of Positional Information in Vision-Language Models (Rösch & Libovický, Findings 2022)

ACL

• Philipp J. Rösch and Jindřich Libovický. 2022. Probing the Role of Positional Information in Vision-Language Models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1031–1041, Seattle, United States. Association for Computational Linguistics.