Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation

Xinyu Pi, Bing Wang, Yan Gao, Jiaqi Guo, Zhoujun Li, Jian-Guang Lou


Abstract
The robustness of Text-to-SQL parsers against adversarial perturbations plays a crucial role in delivering highly reliable applications. Previous studies along this line primarily focused on perturbations in the natural language question side, neglecting the variability of tables. Motivated by this, we propose the Adversarial Table Perturbation (ATP) as a new attacking paradigm to measure robustness of Text-to-SQL models. Following this proposition, we curate ADVETA, the first robustness evaluation benchmark featuring natural and realistic ATPs. All tested state-of-the-art models experience dramatic performance drops on ADVETA, revealing significant room of improvement. To defense against ATP, we build a systematic adversarial training example generation framework tailored for better contextualization of tabular data. Experiments show that our approach brings models best robustness improvement against ATP, while also substantially boost model robustness against NL-side perturbations. We will release ADVETA and code to facilitate future research.
Anthology ID:
2022.acl-long.142
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2007–2022
Language:
URL:
https://aclanthology.org/2022.acl-long.142
DOI:
10.18653/v1/2022.acl-long.142
Bibkey:
Cite (ACL):
Xinyu Pi, Bing Wang, Yan Gao, Jiaqi Guo, Zhoujun Li, and Jian-Guang Lou. 2022. Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2007–2022, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation (Pi et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.142.pdf
Code
 microsoft/ContextualSP
Data
ADVETACoSQLConceptNetMultiNLISParCSpider-RealisticWikiSQL