HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference

Tianyu Liu, Zheng Xin, Baobao Chang, Zhifang Sui


Abstract
Many recent studies have shown that for models trained on datasets for natural language inference (NLI), it is possible to make correct predictions by merely looking at the hypothesis while completely ignoring the premise. In this work, we manage to derive adversarial examples in terms of the hypothesis-only bias and explore eligible ways to mitigate such bias. Specifically, we extract various phrases from the hypotheses (artificial patterns) in the training sets, and show that they have been strong indicators to the specific labels. We then figure out ‘hard’ and ‘easy’ instances from the original test sets whose labels are opposite to or consistent with those indications. We also set up baselines including both pretrained models (BERT, RoBerta, XLNet) and competitive non-pretrained models (InferSent, DAM, ESIM). Apart from the benchmark and baselines, we also investigate two debiasing approaches which exploit the artificial pattern modeling to mitigate such hypothesis-only bias: down-sampling and adversarial training. We believe those methods can be treated as competitive baselines in NLI debiasing tasks.
Anthology ID:
2020.lrec-1.846
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
6852–6860
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.846
DOI:
Bibkey:
Cite (ACL):
Tianyu Liu, Zheng Xin, Baobao Chang, and Zhifang Sui. 2020. HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6852–6860, Marseille, France. European Language Resources Association.
Cite (Informal):
HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference (Liu et al., LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.846.pdf
Data
MultiNLISNLI