Grounding Symbols in Multi-Modal Instructions
Yordan Hristov | Svetlin Penkov | Alex Lascarides | Subramanian Ramamoorthy
Proceedings of the First Workshop on Language Grounding for Robotics
As robots begin to cohabit with humans in semi-structured environments, the need arises to understand instructions involving rich variability—for instance, learning to ground symbols in the physical world. Realistically, this task must cope with small datasets consisting of a particular users’ contextual assignment of meaning to terms. We present a method for processing a raw stream of cross-modal input—i.e., linguistic instructions, visual perception of a scene and a concurrent trace of 3D eye tracking fixations—to produce the segmentation of objects with a correspondent association to high-level concepts. To test our framework we present experiments in a table-top object manipulation scenario. Our results show our model learns the user’s notion of colour and shape from a small number of physical demonstrations, generalising to identifying physical referents for novel combinations of the words.
A Generative Model for User Simulation in a Spatial Navigation Domain
Aciel Eshky | Ben Allison | Subramanian Ramamoorthy | Mark Steedman
Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics
- Yordan Hristov 1
- Svetlin Penkov 1
- Alex Lascarides 1
- Aciel Eshky 1
- Ben Allison 1
- show all...