NavHint: Vision and Language Navigation Agent with a Hint Generator

Yue Zhang, Quan Guo, Parisa Kordjamshidi


Abstract
The existing work on vision and language navigation mainly relies on navigation-related losses to establish the connection between vision and language modalities, neglecting aspects of helping the navigation agent build a deep understanding of the visual environment.In our work, we provide indirect supervision to the navigation agent through a hint generator that provides detailed visual descriptions.The hint generator assists the navigation agent in developing a global understanding of the visual environment. It directs the agent’s attention toward related navigation details, including the relevant sub-instruction, potential challenges in recognition and ambiguities in grounding, and the targeted viewpoint description. To train the hint generator, we construct a synthetic dataset based on landmarks in the instructions and visible and distinctive objects in the visual environment.We evaluate our method on the R2R and R4R datasets and achieve state-of-the-art on several metrics. The experimental results demonstrate that generating hints not only enhances the navigation performance but also helps improve the agent’s interpretability.
Anthology ID:
2024.findings-eacl.7
Volume:
Findings of the Association for Computational Linguistics: EACL 2024
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
92–103
Language:
URL:
https://aclanthology.org/2024.findings-eacl.7
DOI:
Bibkey:
Cite (ACL):
Yue Zhang, Quan Guo, and Parisa Kordjamshidi. 2024. NavHint: Vision and Language Navigation Agent with a Hint Generator. In Findings of the Association for Computational Linguistics: EACL 2024, pages 92–103, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
NavHint: Vision and Language Navigation Agent with a Hint Generator (Zhang et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-eacl.7.pdf
Video:
 https://aclanthology.org/2024.findings-eacl.7.mp4