<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="li-etal-2022-exploring">
    <titleInfo>
        <title>Exploring the GLIDE model for Human Action Effect Prediction</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Fangjun</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="given">C</namePart>
        <namePart type="family">Hogg</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anthony</namePart>
        <namePart type="given">G</namePart>
        <namePart type="family">Cohn</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2nd Workshop on People in Vision, Language, and the Mind</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Patrizia</namePart>
            <namePart type="family">Paggio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Albert</namePart>
            <namePart type="family">Gatt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marc</namePart>
            <namePart type="family">Tanti</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>European Language Resources Association</publisher>
            <place>
                <placeTerm type="text">Marseille, France</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We address the following action-effect prediction task. Given an image depicting an initial state of the world and an action expressed in text, predict an image depicting the state of the world following the action. The prediction should have the same scene context as the input image. We explore the use of the recently proposed GLIDE model for performing this task. GLIDE is a generative neural network that can synthesize (inpaint) masked areas of an image, conditioned on a short piece of text. Our idea is to mask-out a region of the input image where the effect of the action is expected to occur. GLIDE is then used to inpaint the masked region conditioned on the required action. In this way, the resulting image has the same background context as the input image, updated to show the effect of the action. We give qualitative results from experiments using the EPIC dataset of ego-centric videos labelled with actions.</abstract>
    <identifier type="citekey">li-etal-2022-exploring</identifier>
    <location>
        <url>https://aclanthology.org/2022.pvlam-1.1</url>
    </location>
    <part>
        <date>2022-06</date>
        <extent unit="page">
            <start>1</start>
            <end>5</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Exploring the GLIDE model for Human Action Effect Prediction
%A Li, Fangjun
%A Hogg, David C.
%A Cohn, Anthony G.
%Y Paggio, Patrizia
%Y Gatt, Albert
%Y Tanti, Marc
%S Proceedings of the 2nd Workshop on People in Vision, Language, and the Mind
%D 2022
%8 June
%I European Language Resources Association
%C Marseille, France
%F li-etal-2022-exploring
%X We address the following action-effect prediction task. Given an image depicting an initial state of the world and an action expressed in text, predict an image depicting the state of the world following the action. The prediction should have the same scene context as the input image. We explore the use of the recently proposed GLIDE model for performing this task. GLIDE is a generative neural network that can synthesize (inpaint) masked areas of an image, conditioned on a short piece of text. Our idea is to mask-out a region of the input image where the effect of the action is expected to occur. GLIDE is then used to inpaint the masked region conditioned on the required action. In this way, the resulting image has the same background context as the input image, updated to show the effect of the action. We give qualitative results from experiments using the EPIC dataset of ego-centric videos labelled with actions.
%U https://aclanthology.org/2022.pvlam-1.1
%P 1-5

Markdown (Informal)

[Exploring the GLIDE model for Human Action Effect Prediction](https://aclanthology.org/2022.pvlam-1.1) (Li et al., PVLAM 2022)

• Exploring the GLIDE model for Human Action Effect Prediction (Li et al., PVLAM 2022)

ACL

• Fangjun Li, David C. Hogg, and Anthony G. Cohn. 2022. Exploring the GLIDE model for Human Action Effect Prediction. In Proceedings of the 2nd Workshop on People in Vision, Language, and the Mind, pages 1–5, Marseille, France. European Language Resources Association.