<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="min-etal-2022-dont">
    <titleInfo>
        <title>Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">So</namePart>
        <namePart type="given">Yeon</namePart>
        <namePart type="family">Min</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hao</namePart>
        <namePart type="family">Zhu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ruslan</namePart>
        <namePart type="family">Salakhutdinov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yonatan</namePart>
        <namePart type="family">Bisk</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yoav</namePart>
            <namePart type="family">Goldberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yue</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. The recent introduction of benchmarks raises the question of how best to train and evaluate models for this multi-turn, multi-agent, long-horizon task. This paper contributes to that conversation, by arguing that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dialogue research and may hinder progress. We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress. First, we observe that models trained with IL take spurious actions during evaluation. Second, we find that existing models fail to ground query utterances, which are essential for task completion. Third, we argue evaluation should focus on higher-level semantic goals. We will release code to additionally filter the data and benchmark models for improved evaluation.</abstract>
    <identifier type="citekey">min-etal-2022-dont</identifier>
    <identifier type="doi">10.18653/v1/2022.emnlp-main.635</identifier>
    <location>
        <url>https://aclanthology.org/2022.emnlp-main.635</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>9361</start>
            <end>9368</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue
%A Min, So Yeon
%A Zhu, Hao
%A Salakhutdinov, Ruslan
%A Bisk, Yonatan
%Y Goldberg, Yoav
%Y Kozareva, Zornitsa
%Y Zhang, Yue
%S Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates
%F min-etal-2022-dont
%X Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. The recent introduction of benchmarks raises the question of how best to train and evaluate models for this multi-turn, multi-agent, long-horizon task. This paper contributes to that conversation, by arguing that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dialogue research and may hinder progress. We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress. First, we observe that models trained with IL take spurious actions during evaluation. Second, we find that existing models fail to ground query utterances, which are essential for task completion. Third, we argue evaluation should focus on higher-level semantic goals. We will release code to additionally filter the data and benchmark models for improved evaluation.
%R 10.18653/v1/2022.emnlp-main.635
%U https://aclanthology.org/2022.emnlp-main.635
%U https://doi.org/10.18653/v1/2022.emnlp-main.635
%P 9361-9368

Markdown (Informal)

[Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue](https://aclanthology.org/2022.emnlp-main.635) (Min et al., EMNLP 2022)

• Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue (Min et al., EMNLP 2022)

ACL

• So Yeon Min, Hao Zhu, Ruslan Salakhutdinov, and Yonatan Bisk. 2022. Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9361–9368, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.