Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas

Raphael Schumann, Stefan Riezler


Abstract
Vision and language navigation (VLN) is a challenging visually-grounded language understanding task. Given a natural language navigation instruction, a visual agent interacts with a graph-based environment equipped with panorama images and tries to follow the described route. Most prior work has been conducted in indoor scenarios where best results were obtained for navigation on routes that are similar to the training routes, with sharp drops in performance when testing on unseen environments. We focus on VLN in outdoor scenarios and find that in contrast to indoor VLN, most of the gain in outdoor VLN on unseen data is due to features like junction type embedding or heading delta that are specific to the respective environment graph, while image information plays a very minor role in generalizing VLN to unseen outdoor areas. These findings show a bias to specifics of graph representations of urban environments, demanding that VLN tasks grow in scale and diversity of geographical environments.
Anthology ID:
2022.acl-long.518
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7519–7532
Language:
URL:
https://aclanthology.org/2022.acl-long.518
DOI:
10.18653/v1/2022.acl-long.518
Bibkey:
Cite (ACL):
Raphael Schumann and Stefan Riezler. 2022. Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7519–7532, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas (Schumann & Riezler, ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.518.pdf
Software:
 2022.acl-long.518.software.zip
Video:
 https://aclanthology.org/2022.acl-long.518.mp4
Code
 raphael-sch/map2seq_vln
Data
Touchdown Datasetmap2seq