Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach

Daichi Haraguchi, Kiyoaki Shirai, Naoya Inoue, Natthawut Kertkeidkachorn


Abstract
Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model. While a number of previous work has tackled the identification of shortcut reasoning, there are still two major limitations: (i) a method for quantifying the severity of the discovered shortcut reasoning is not provided; (ii) certain types of shortcut reasoning may be missed. To address these issues, we propose a novel method for identifying shortcut reasoning. The proposed method quantifies the severity of the shortcut reasoning by leveraging out-of-distribution data and does not make any assumptions about the type of tokens triggering the shortcut reasoning. Our experiments on Natural Language Inference and Sentiment Analysis demonstrate that our framework successfully discovers known and unknown shortcut reasoning in the previous work.
Anthology ID:
2023.findings-emnlp.424
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6401–6407
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.424
DOI:
10.18653/v1/2023.findings-emnlp.424
Bibkey:
Cite (ACL):
Daichi Haraguchi, Kiyoaki Shirai, Naoya Inoue, and Natthawut Kertkeidkachorn. 2023. Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6401–6407, Singapore. Association for Computational Linguistics.
Cite (Informal):
Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach (Haraguchi et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-emnlp.424.pdf