Grounding Meaning Representation for Situated Reasoning

Nikhil Krishnaswamy, James Pustejovsky


Abstract
As natural language technology becomes ever-present in everyday life, people will expect artificial agents to understand language use as humans do. Nevertheless, most advanced neural AI systems fail at some types of interactions that are trivial for humans (e.g., ask a smart system “What am I pointing at?”). One critical aspect of human language understanding is situated reasoning, where inferences make reference to the local context, perceptual surroundings, and contextual groundings from the interaction. In this cutting-edge tutorial, we bring to the NLP/CL community a synthesis of multimodal grounding and meaning representation techniques with formal and computational models of embodied reasoning. We will discuss existing approaches to multimodal language grounding and meaning representations, discuss the kind of information each method captures and their relative suitability to situated reasoning tasks, and demon- strate how to construct agents that conduct situated reasoning by embodying a simulated environment. In doing so, these agents also represent their human interlocutor(s) within the simulation, and are represented through their virtual embodiment in the real world, enabling true bidirectional communication with a computer using multiple modalities.
Anthology ID:
2022.aacl-tutorials.4
Volume:
Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Tutorial Abstracts
Month:
November
Year:
2022
Address:
Taipei
Editors:
Miguel A. Alonso, Zhongyu Wei
Venues:
AACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
22–27
Language:
URL:
https://aclanthology.org/2022.aacl-tutorials.4
DOI:
10.18653/v1/2022.aacl-tutorials.4
Bibkey:
Cite (ACL):
Nikhil Krishnaswamy and James Pustejovsky. 2022. Grounding Meaning Representation for Situated Reasoning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Tutorial Abstracts, pages 22–27, Taipei. Association for Computational Linguistics.
Cite (Informal):
Grounding Meaning Representation for Situated Reasoning (Krishnaswamy & Pustejovsky, AACL-IJCNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.aacl-tutorials.4.pdf