<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhou-etal-2023-sequence">
    <titleInfo>
        <title>Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xiang</namePart>
        <namePart type="family">Zhou</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Aditya</namePart>
        <namePart type="family">Gupta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shyam</namePart>
        <namePart type="family">Upadhyay</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mohit</namePart>
        <namePart type="family">Bansal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Manaal</namePart>
        <namePart type="family">Faruqui</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Alexis</namePart>
            <namePart type="family">Palmer</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jose</namePart>
            <namePart type="family">Camacho-collados</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>While many real-life tasks require reasoning over multi-step sequential instructions, collecting fine-grained annotations for each intermediate step can be prohibitively expensive. In this work, we study how general pretrained sequence-to-sequence transformers perform under varying types of annotation for sequential instruction understanding. We conduct experiments using T5 (Raffel et al., 2020) on a commonly-used multi-step instruction understanding dataset SCONE (Long et al., 2016) that includes three sub-tasks. First, we show that with only gold supervision for the final step of a multi-step instruction sequence, depending on the sequential properties of different tasks, transformers may exhibit extremely bad performance on intermediate steps, in stark contrast with their performance on the final step. Next, we explore two directions to relieve this problem. We show that with the same limited annotation budget, using supervision uniformly distributed across different steps (instead of only final-step supervision), we can greatly improve the performance on intermediate steps with a drop in final-step performance. Further, we explore a contrastive learning approach to provide training signals on intermediate steps with zero intermediate gold supervision. This, however, achieves mixed results. It significantly improves the model’s bad intermediate-step performance on one subtask, but also shows decreased performance on another subtask.</abstract>
    <identifier type="citekey">zhou-etal-2023-sequence</identifier>
    <identifier type="doi">10.18653/v1/2023.starsem-1.45</identifier>
    <location>
        <url>https://aclanthology.org/2023.starsem-1.45</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>527</start>
            <end>534</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions?
%A Zhou, Xiang
%A Gupta, Aditya
%A Upadhyay, Shyam
%A Bansal, Mohit
%A Faruqui, Manaal
%Y Palmer, Alexis
%Y Camacho-collados, Jose
%S Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F zhou-etal-2023-sequence
%X While many real-life tasks require reasoning over multi-step sequential instructions, collecting fine-grained annotations for each intermediate step can be prohibitively expensive. In this work, we study how general pretrained sequence-to-sequence transformers perform under varying types of annotation for sequential instruction understanding. We conduct experiments using T5 (Raffel et al., 2020) on a commonly-used multi-step instruction understanding dataset SCONE (Long et al., 2016) that includes three sub-tasks. First, we show that with only gold supervision for the final step of a multi-step instruction sequence, depending on the sequential properties of different tasks, transformers may exhibit extremely bad performance on intermediate steps, in stark contrast with their performance on the final step. Next, we explore two directions to relieve this problem. We show that with the same limited annotation budget, using supervision uniformly distributed across different steps (instead of only final-step supervision), we can greatly improve the performance on intermediate steps with a drop in final-step performance. Further, we explore a contrastive learning approach to provide training signals on intermediate steps with zero intermediate gold supervision. This, however, achieves mixed results. It significantly improves the model’s bad intermediate-step performance on one subtask, but also shows decreased performance on another subtask.
%R 10.18653/v1/2023.starsem-1.45
%U https://aclanthology.org/2023.starsem-1.45
%U https://doi.org/10.18653/v1/2023.starsem-1.45
%P 527-534

Markdown (Informal)

[Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions?](https://aclanthology.org/2023.starsem-1.45) (Zhou et al., *SEM 2023)

• Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions? (Zhou et al., *SEM 2023)

ACL

• Xiang Zhou, Aditya Gupta, Shyam Upadhyay, Mohit Bansal, and Manaal Faruqui. 2023. Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions?. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 527–534, Toronto, Canada. Association for Computational Linguistics.