Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments

Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, Angel Chang


Abstract
In the Vision-and-Language Navigation (VLN) task an embodied agent navigates a 3D environment, following natural language instructions. A challenge in this task is how to handle ‘off the path’ scenarios where an agent veers from a reference path. Prior work supervises the agent with actions based on the shortest path from the agent’s location to the goal, but such goal-oriented supervision is often not in alignment with the instruction. Furthermore, the evaluation metrics employed by prior work do not measure how much of a language instruction the agent is able to follow. In this work, we propose a simple and effective language-aligned supervision scheme, and a new metric that measures the number of sub-instructions the agent has completed during navigation.
Anthology ID:
2021.emnlp-main.328
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4018–4028
Language:
URL:
https://aclanthology.org/2021.emnlp-main.328
DOI:
10.18653/v1/2021.emnlp-main.328
Bibkey:
Cite (ACL):
Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, and Angel Chang. 2021. Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4018–4028, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments (Raychaudhuri et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.328.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.328.mp4