Guided K-best Selection for Semantic Parsing Annotation

Anton Belyy, Chieh-yang Huang, Jacob Andreas, Emmanouil Antonios Platanios, Sam Thomson, Richard Shin, Subhro Roy, Aleksandr Nisnevich, Charles Chen, Benjamin Van Durme


Abstract
Collecting data for conversational semantic parsing is a time-consuming and demanding process. In this paper we consider, given an incomplete dataset with only a small amount of data, how to build an AI-powered human-in-the-loop process to enable efficient data collection. A guided K-best selection process is proposed, which (i) generates a set of possible valid candidates; (ii) allows users to quickly traverse the set and filter incorrect parses; and (iii) asks users to select the correct parse, with minimal modification when necessary. We investigate how to best support users in efficiently traversing the candidate set and locating the correct parse, in terms of speed and accuracy. In our user study, consisting of five annotators labeling 300 instances each, we find that combining keyword searching, where keywords can be used to query relevant candidates, and keyword suggestion, where representative keywords are automatically generated, enables fast and accurate annotation.
Anthology ID:
2022.acl-demo.11
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
114–126
Language:
URL:
https://aclanthology.org/2022.acl-demo.11
DOI:
10.18653/v1/2022.acl-demo.11
Bibkey:
Cite (ACL):
Anton Belyy, Chieh-yang Huang, Jacob Andreas, Emmanouil Antonios Platanios, Sam Thomson, Richard Shin, Subhro Roy, Aleksandr Nisnevich, Charles Chen, and Benjamin Van Durme. 2022. Guided K-best Selection for Semantic Parsing Annotation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 114–126, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Guided K-best Selection for Semantic Parsing Annotation (Belyy et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-demo.11.pdf