longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks.

Venelin Kovatchev, Trina Chatterjee, Venkata S Govindarajan, Jifan Chen, Eunsol Choi, Gabriella Chronis, Anubrata Das, Katrin Erk, Matthew Lease, Junyi Jessy Li, Yating Wu, Kyle Mahowald


Abstract
Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team “longhorns” on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first (pending validation), with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.
Anthology ID:
2022.dadc-1.5
Volume:
Proceedings of the First Workshop on Dynamic Adversarial Data Collection
Month:
July
Year:
2022
Address:
Seattle, WA
Editors:
Max Bartolo, Hannah Kirk, Pedro Rodriguez, Katerina Margatina, Tristan Thrush, Robin Jia, Pontus Stenetorp, Adina Williams, Douwe Kiela
Venue:
DADC
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
41–52
Language:
URL:
https://aclanthology.org/2022.dadc-1.5
DOI:
10.18653/v1/2022.dadc-1.5
Bibkey:
Cite (ACL):
Venelin Kovatchev, Trina Chatterjee, Venkata S Govindarajan, Jifan Chen, Eunsol Choi, Gabriella Chronis, Anubrata Das, Katrin Erk, Matthew Lease, Junyi Jessy Li, Yating Wu, and Kyle Mahowald. 2022. longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks.. In Proceedings of the First Workshop on Dynamic Adversarial Data Collection, pages 41–52, Seattle, WA. Association for Computational Linguistics.
Cite (Informal):
longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. (Kovatchev et al., DADC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.dadc-1.5.pdf