VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator

Ayush Shrivastava, Karthik Gopalakrishnan, Yang Liu, Robinson Piramuthu, Gokhan Tur, Devi Parikh, Dilek Hakkani-Tur


Abstract
Interactive robots navigating photo-realistic environments need to be trained to effectively leverage and handle the dynamic nature of dialogue in addition to the challenges underlying vision-and-language navigation (VLN). In this paper, we present VISITRON, a multi-modal Transformer-based navigator better suited to the interactive regime inherent to Cooperative Vision-and-Dialog Navigation (CVDN). VISITRON is trained to: i) identify and associate object-level concepts and semantics between the environment and dialogue history, ii) identify when to interact vs. navigate via imitation learning of a binary classification head. We perform extensive pre-training and fine-tuning ablations with VISITRON to gain empirical insights and improve performance on CVDN. VISITRON’s ability to identify when to interact leads to a natural generalization of the game-play mode introduced by Roman et al. (2020) for enabling the use of such models in different environments. VISITRON is competitive with models on the static CVDN leaderboard and attains state-of-the-art performance on the Success weighted by Path Length (SPL) metric.
Anthology ID:
2022.findings-acl.157
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1984–1994
Language:
URL:
https://aclanthology.org/2022.findings-acl.157
DOI:
10.18653/v1/2022.findings-acl.157
Bibkey:
Cite (ACL):
Ayush Shrivastava, Karthik Gopalakrishnan, Yang Liu, Robinson Piramuthu, Gokhan Tur, Devi Parikh, and Dilek Hakkani-Tur. 2022. VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1984–1994, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator (Shrivastava et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.157.pdf
Video:
 https://aclanthology.org/2022.findings-acl.157.mp4
Code
 alexa/visitron
Data
Matterport3DRxR