Apple Core-dination: Linguistic Feedback and Learning in a Speech-to-Action Shared World Game

Susann Boy, AriaRay Brown, Morgan Wixted


Abstract
We investigate the question of how adaptive feedback from a virtual agent impacts the linguistic input of the user in a shared world game environment. To do so, we carry out an exploratory pilot study to observe how individualized linguistic feedback affects the user’s speech input. We introduce a speech-controlled game, Apple Core-dination, in which an agent learns complex tasks using a base knowledge of simple actions. The agent is equipped with a learning mechanism for mapping new commands to sequences of simple actions, as well as the ability to incorporate user input into written responses. The agent repeatedly shares its internal knowledge state by responding to what it knows and does not know about language meaning and the shared environment. Our paper focuses on the linguistic feedback loop in order to analyze the nature of user input. Feedback from the agent is provided in the form of visual movement and written linguistic responses. Particular attention is given to incorporating user input into agent responses and updating the speech-to-action mappings based on commands provided by the user. Through our pilot study, we analyze task success and compare the lexical features of user input. Results show variation in input length and lexical variety across users, suggesting a correlation between the two that can be studied further.
Anthology ID:
2021.internlp-1.2
Volume:
Proceedings of the First Workshop on Interactive Learning for Natural Language Processing
Month:
August
Year:
2021
Address:
Online
Editors:
Kianté Brantley, Soham Dan, Iryna Gurevych, Ji-Ung Lee, Filip Radlinski, Hinrich Schütze, Edwin Simpson, Lili Yu
Venue:
InterNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7–15
Language:
URL:
https://aclanthology.org/2021.internlp-1.2
DOI:
10.18653/v1/2021.internlp-1.2
Bibkey:
Cite (ACL):
Susann Boy, AriaRay Brown, and Morgan Wixted. 2021. Apple Core-dination: Linguistic Feedback and Learning in a Speech-to-Action Shared World Game. In Proceedings of the First Workshop on Interactive Learning for Natural Language Processing, pages 7–15, Online. Association for Computational Linguistics.
Cite (Informal):
Apple Core-dination: Linguistic Feedback and Learning in a Speech-to-Action Shared World Game (Boy et al., InterNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.internlp-1.2.pdf