<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bhattacharyya-etal-2023-crapes">
    <titleInfo>
        <title>CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Abhidip</namePart>
        <namePart type="family">Bhattacharyya</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Martha</namePart>
        <namePart type="family">Palmer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Christoffer</namePart>
        <namePart type="family">Heckman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Alexis</namePart>
            <namePart type="family">Palmer</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jose</namePart>
            <namePart type="family">Camacho-collados</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Automatic image comprehension is an important yet challenging task that includes identifying actions in an image and corresponding action participants. Most current approaches to this task, now termed Grounded Situation Recognition (GSR), start by predicting a verb that describes the action and then predict the nouns that can participate in the action as arguments to the verb. This problem formulation limits each image to a single action even though several actions could be depicted. In contrast, text-based Semantic Role Labeling (SRL) aims to label all actions in a sentence, typically resulting in at least two or three predicate argument structures per sentence. We hypothesize that expanding GSR to follow the more liberal SRL text-based approach to action and participant identification could improve image comprehension results. To test this hypothesis and to preserve generalization capabilities, we use general-purpose vision and language components as a front-end. This paper presents our results, a substantial 28.6 point jump in performance on the SWiG dataset, which confirm our hypothesis. We also discuss the benefits of loosely coupled broad-coverage off-the-shelf components which generalized well to out of domain images, and can decrease the need for manual image semantic role annotation.</abstract>
    <identifier type="citekey">bhattacharyya-etal-2023-crapes</identifier>
    <identifier type="doi">10.18653/v1/2023.starsem-1.7</identifier>
    <location>
        <url>https://aclanthology.org/2023.starsem-1.7</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>61</start>
            <end>70</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling
%A Bhattacharyya, Abhidip
%A Palmer, Martha
%A Heckman, Christoffer
%Y Palmer, Alexis
%Y Camacho-collados, Jose
%S Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F bhattacharyya-etal-2023-crapes
%X Automatic image comprehension is an important yet challenging task that includes identifying actions in an image and corresponding action participants. Most current approaches to this task, now termed Grounded Situation Recognition (GSR), start by predicting a verb that describes the action and then predict the nouns that can participate in the action as arguments to the verb. This problem formulation limits each image to a single action even though several actions could be depicted. In contrast, text-based Semantic Role Labeling (SRL) aims to label all actions in a sentence, typically resulting in at least two or three predicate argument structures per sentence. We hypothesize that expanding GSR to follow the more liberal SRL text-based approach to action and participant identification could improve image comprehension results. To test this hypothesis and to preserve generalization capabilities, we use general-purpose vision and language components as a front-end. This paper presents our results, a substantial 28.6 point jump in performance on the SWiG dataset, which confirm our hypothesis. We also discuss the benefits of loosely coupled broad-coverage off-the-shelf components which generalized well to out of domain images, and can decrease the need for manual image semantic role annotation.
%R 10.18653/v1/2023.starsem-1.7
%U https://aclanthology.org/2023.starsem-1.7
%U https://doi.org/10.18653/v1/2023.starsem-1.7
%P 61-70

Markdown (Informal)

[CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling](https://aclanthology.org/2023.starsem-1.7) (Bhattacharyya et al., *SEM 2023)

• CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling (Bhattacharyya et al., *SEM 2023)

ACL

• Abhidip Bhattacharyya, Martha Palmer, and Christoffer Heckman. 2023. CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 61–70, Toronto, Canada. Association for Computational Linguistics.