<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="li-etal-2020-interactive">
    <titleInfo>
        <title>Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Toby</namePart>
        <namePart type="given">Jia-Jun</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tom</namePart>
        <namePart type="family">Mitchell</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Brad</namePart>
        <namePart type="family">Myers</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Asli</namePart>
            <namePart type="family">Celikyilmaz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tsung-Hsien</namePart>
            <namePart type="family">Wen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We show SUGILITE, an intelligent task automation agent that can learn new tasks and relevant associated concepts interactively from the user’s natural language instructions and demonstrations, using the graphical user interfaces (GUIs) of third-party mobile apps. This system provides several interesting features: (1) it allows users to teach new task procedures and concepts through verbal instructions together with demonstration of the steps of a script using GUIs; (2) it supports users in clarifying their intents for demonstrated actions using GUI-grounded verbal instructions; (3) it infers parameters of tasks and their possible values in utterances using the hierarchical structures of the underlying app GUIs; and (4) it generalizes taught concepts to different contexts and task domains. We describe the architecture of the SUGILITE system, explain the design and implementation of its key features, and show a prototype in the form of a conversational assistant on Android.</abstract>
    <identifier type="citekey">li-etal-2020-interactive</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-demos.25</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-demos.25</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>215</start>
            <end>223</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations
%A Li, Toby Jia-Jun
%A Mitchell, Tom
%A Myers, Brad
%Y Celikyilmaz, Asli
%Y Wen, Tsung-Hsien
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F li-etal-2020-interactive
%X We show SUGILITE, an intelligent task automation agent that can learn new tasks and relevant associated concepts interactively from the user’s natural language instructions and demonstrations, using the graphical user interfaces (GUIs) of third-party mobile apps. This system provides several interesting features: (1) it allows users to teach new task procedures and concepts through verbal instructions together with demonstration of the steps of a script using GUIs; (2) it supports users in clarifying their intents for demonstrated actions using GUI-grounded verbal instructions; (3) it infers parameters of tasks and their possible values in utterances using the hierarchical structures of the underlying app GUIs; and (4) it generalizes taught concepts to different contexts and task domains. We describe the architecture of the SUGILITE system, explain the design and implementation of its key features, and show a prototype in the form of a conversational assistant on Android.
%R 10.18653/v1/2020.acl-demos.25
%U https://aclanthology.org/2020.acl-demos.25
%U https://doi.org/10.18653/v1/2020.acl-demos.25
%P 215-223

Markdown (Informal)

[Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations](https://aclanthology.org/2020.acl-demos.25) (Li et al., ACL 2020)

• Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations (Li et al., ACL 2020)

ACL

• Toby Jia-Jun Li, Tom Mitchell, and Brad Myers. 2020. Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 215–223, Online. Association for Computational Linguistics.