Interactive Symbol Grounding with Complex Referential Expressions

Rimvydas Rubavicius, Alex Lascarides


Abstract
We present a procedure for learning to ground symbols from a sequence of stimuli consisting of an arbitrarily complex noun phrase (e.g. “all but one green square above both red circles.”) and its designation in the visual scene. Our distinctive approach combines: a) lazy few-shot learning to relate open-class words like green and above to their visual percepts; and b) symbolic reasoning with closed-class word categories like quantifiers and negation. We use this combination to estimate new training examples for grounding symbols that occur within a noun phrase but aren’t designated by that noun phase (e.g, red in the above example), thereby potentially gaining data efficiency. We evaluate the approach in a visual reference resolution task, in which the learner starts out unaware of concepts that are part of the domain model and how they relate to visual percepts.
Anthology ID:
2022.naacl-main.358
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4863–4874
Language:
URL:
https://aclanthology.org/2022.naacl-main.358
DOI:
10.18653/v1/2022.naacl-main.358
Bibkey:
Cite (ACL):
Rimvydas Rubavicius and Alex Lascarides. 2022. Interactive Symbol Grounding with Complex Referential Expressions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4863–4874, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Interactive Symbol Grounding with Complex Referential Expressions (Rubavicius & Lascarides, NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.358.pdf
Software:
 2022.naacl-main.358.software.zip
Video:
 https://aclanthology.org/2022.naacl-main.358.mp4
Code
 itl-ed/igre