<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yun-etal-2023-emergence">
    <titleInfo>
        <title>Emergence of Abstract State Representations in Embodied Sequence Modeling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tian</namePart>
        <namePart type="family">Yun</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zilai</namePart>
        <namePart type="family">Zeng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kunal</namePart>
        <namePart type="family">Handa</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ashish</namePart>
        <namePart type="family">Thapliyal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Bo</namePart>
        <namePart type="family">Pang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ellie</namePart>
        <namePart type="family">Pavlick</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Chen</namePart>
        <namePart type="family">Sun</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Decision making via sequence modeling aims to mimic the success of language models, where actions taken by an embodied agent are modeled as tokens to predict. Despite their promising performance, it remains unclear if embodied sequence modeling leads to the emergence of internal representations that represent the environmental state information. A model that lacks abstract state representations would be liable to make decisions based on surface statistics which fail to generalize. We take the BabyAI environment, a grid world in which language-conditioned navigation tasks are performed, and build a sequence modeling Transformer, which takes a language instruction, a sequence of actions, and environmental observations as its inputs. In order to investigate the emergence of abstract state representations, we design a “blindfolded” navigation task, where only the initial environmental layout, the language instruction, and the action sequence to complete the task are available for training. Our probing results show that intermediate environmental layouts can be reasonably reconstructed from the internal activations of a trained model, and that language instructions play a role in the reconstruction accuracy. Our results suggest that many key features of state representations can emerge via embodied sequence modeling, supporting an optimistic outlook for applications of sequence modeling objectives to more complex embodied decision-making domains.</abstract>
    <identifier type="citekey">yun-etal-2023-emergence</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.749</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.749</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>12190</start>
            <end>12205</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Emergence of Abstract State Representations in Embodied Sequence Modeling
%A Yun, Tian
%A Zeng, Zilai
%A Handa, Kunal
%A Thapliyal, Ashish
%A Pang, Bo
%A Pavlick, Ellie
%A Sun, Chen
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F yun-etal-2023-emergence
%X Decision making via sequence modeling aims to mimic the success of language models, where actions taken by an embodied agent are modeled as tokens to predict. Despite their promising performance, it remains unclear if embodied sequence modeling leads to the emergence of internal representations that represent the environmental state information. A model that lacks abstract state representations would be liable to make decisions based on surface statistics which fail to generalize. We take the BabyAI environment, a grid world in which language-conditioned navigation tasks are performed, and build a sequence modeling Transformer, which takes a language instruction, a sequence of actions, and environmental observations as its inputs. In order to investigate the emergence of abstract state representations, we design a “blindfolded” navigation task, where only the initial environmental layout, the language instruction, and the action sequence to complete the task are available for training. Our probing results show that intermediate environmental layouts can be reasonably reconstructed from the internal activations of a trained model, and that language instructions play a role in the reconstruction accuracy. Our results suggest that many key features of state representations can emerge via embodied sequence modeling, supporting an optimistic outlook for applications of sequence modeling objectives to more complex embodied decision-making domains.
%R 10.18653/v1/2023.emnlp-main.749
%U https://aclanthology.org/2023.emnlp-main.749
%U https://doi.org/10.18653/v1/2023.emnlp-main.749
%P 12190-12205

Markdown (Informal)

[Emergence of Abstract State Representations in Embodied Sequence Modeling](https://aclanthology.org/2023.emnlp-main.749) (Yun et al., EMNLP 2023)

• Emergence of Abstract State Representations in Embodied Sequence Modeling (Yun et al., EMNLP 2023)

ACL

• Tian Yun, Zilai Zeng, Kunal Handa, Ashish Thapliyal, Bo Pang, Ellie Pavlick, and Chen Sun. 2023. Emergence of Abstract State Representations in Embodied Sequence Modeling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12190–12205, Singapore. Association for Computational Linguistics.