How Fast can BERT Learn Simple Natural Language Inference?

Yi-Chung Lin, Keh-Yih Su


Abstract
This paper empirically studies whether BERT can really learn to conduct natural language inference (NLI) without utilizing hidden dataset bias; and how efficiently it can learn if it could. This is done via creating a simple entailment judgment case which involves only binary predicates in plain English. The results show that the learning process of BERT is very slow. However, the efficiency of learning can be greatly improved (data reduction by a factor of 1,500) if task-related features are added. This suggests that domain knowledge greatly helps when conducting NLI with neural networks.
Anthology ID:
2021.eacl-main.51
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Editors:
Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
626–633
Language:
URL:
https://aclanthology.org/2021.eacl-main.51
DOI:
10.18653/v1/2021.eacl-main.51
Bibkey:
Cite (ACL):
Yi-Chung Lin and Keh-Yih Su. 2021. How Fast can BERT Learn Simple Natural Language Inference?. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 626–633, Online. Association for Computational Linguistics.
Cite (Informal):
How Fast can BERT Learn Simple Natural Language Inference? (Lin & Su, EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.51.pdf
Software:
 2021.eacl-main.51.Software.txt
Data
SNLI