Challenges in Designing Natural Language Interfaces for Complex Visual Models

Henrik Voigt, Monique Meuschke, Kai Lawonn, Sina Zarrieß


Abstract
Intuitive interaction with visual models becomes an increasingly important task in the field of Visualization (VIS) and verbal interaction represents a significant aspect of it. Vice versa, modeling verbal interaction in visual environments is a major trend in ongoing research in NLP. To date, research on Language & Vision, however, mostly happens at the intersection of NLP and Computer Vision (CV), and much less at the intersection of NLP and Visualization, which is an important area in Human-Computer Interaction (HCI). This paper presents a brief survey of recent work on interactive tasks and set-ups in NLP and Visualization. We discuss the respective methods, show interesting gaps, and conclude by suggesting neural, visually grounded dialogue modeling as a promising potential for NLIs for visual models.
Anthology ID:
2021.hcinlp-1.11
Volume:
Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing
Month:
April
Year:
2021
Address:
Online
Editors:
Su Lin Blodgett, Michael Madaio, Brendan O'Connor, Hanna Wallach, Qian Yang
Venue:
HCINLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
66–73
Language:
URL:
https://aclanthology.org/2021.hcinlp-1.11
DOI:
Bibkey:
Cite (ACL):
Henrik Voigt, Monique Meuschke, Kai Lawonn, and Sina Zarrieß. 2021. Challenges in Designing Natural Language Interfaces for Complex Visual Models. In Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, pages 66–73, Online. Association for Computational Linguistics.
Cite (Informal):
Challenges in Designing Natural Language Interfaces for Complex Visual Models (Voigt et al., HCINLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.hcinlp-1.11.pdf
Data
Visual Question Answering