Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing

Alane Suhr, Ming-Wei Chang, Peter Shaw, Kenton Lee


Abstract
We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, including Spider, were proposed to support development of XSP systems. We propose a challenging evaluation setup for cross-database semantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic parsing, demonstrating the need to use and develop diverse training and evaluation datasets.
Anthology ID:
2020.acl-main.742
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8372–8388
Language:
URL:
https://aclanthology.org/2020.acl-main.742
DOI:
10.18653/v1/2020.acl-main.742
Bibkey:
Cite (ACL):
Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8372–8388, Online. Association for Computational Linguistics.
Cite (Informal):
Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing (Suhr et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.742.pdf
Video:
 http://slideslive.com/38929266