Fast Few-shot Debugging for NLU Test Suites

Christopher Malon, Kai Li, Erik Kruus


Abstract
We study few-shot debugging of transformer based natural language understanding models, using recently popularized test suites to not just diagnose but correct a problem. Given a few debugging examples of a certain phenomenon, and a held-out test set of the same phenomenon, we aim to maximize accuracy on the phenomenon at a minimal cost of accuracy on the original test set. We examine several methods that are faster than full epoch retraining. We introduce a new fast method, which samples a few in-danger examples from the original training set. Compared to fast methods using parameter distance constraints or Kullback-Leibler divergence, we achieve superior original accuracy for comparable debugging accuracy.
Anthology ID:
2022.deelio-1.8
Volume:
Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
Month:
May
Year:
2022
Address:
Dublin, Ireland and Online
Editors:
Eneko Agirre, Marianna Apidianaki, Ivan Vulić
Venue:
DeeLIO
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
79–86
Language:
URL:
https://aclanthology.org/2022.deelio-1.8
DOI:
10.18653/v1/2022.deelio-1.8
Bibkey:
Cite (ACL):
Christopher Malon, Kai Li, and Erik Kruus. 2022. Fast Few-shot Debugging for NLU Test Suites. In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 79–86, Dublin, Ireland and Online. Association for Computational Linguistics.
Cite (Informal):
Fast Few-shot Debugging for NLU Test Suites (Malon et al., DeeLIO 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.deelio-1.8.pdf
Video:
 https://aclanthology.org/2022.deelio-1.8.mp4
Code
 necla-ml/debug-test-suites
Data
GLUEMultiNLISSTSST-2