Detecting Compositionally Out-of-Distribution Examples in Semantic Parsing

Denis Lukovnikov, Sina Daubener, Asja Fischer


Abstract
While neural networks are ubiquitous in state-of-the-art semantic parsers, it has been shown that most standard models suffer from dramatic performance losses when faced with compositionally out-of-distribution (OOD) data. Recently several methods have been proposed to improve compositional generalization in semantic parsing. In this work we instead focus on the problem of detecting compositionally OOD examples with neural semantic parsers, which, to the best of our knowledge, has not been investigated before. We investigate several strong yet simple methods for OOD detection based on predictive uncertainty. The experimental results demonstrate that these techniques perform well on the standard SCAN and CFQ datasets. Moreover, we show that OOD detection can be further improved by using a heterogeneous ensemble.
Anthology ID:
2021.findings-emnlp.54
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
591–598
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.54
DOI:
10.18653/v1/2021.findings-emnlp.54
Bibkey:
Cite (ACL):
Denis Lukovnikov, Sina Daubener, and Asja Fischer. 2021. Detecting Compositionally Out-of-Distribution Examples in Semantic Parsing. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 591–598, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Detecting Compositionally Out-of-Distribution Examples in Semantic Parsing (Lukovnikov et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.54.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.54.mp4
Data
CFQSCAN