Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks

Carlos Aspillaga, Andrés Carvallo, Vladimir Araujo


Abstract
There has been significant progress in recent years in the field of Natural Language Processing thanks to the introduction of the Transformer architecture. Current state-of-the-art models, via a large number of parameters and pre-training on massive text corpus, have shown impressive results on several downstream tasks. Many researchers have studied previous (non-Transformer) models to understand their actual behavior under different scenarios, showing that these models are taking advantage of clues or failures of datasets and that slight perturbations on the input data can severely reduce their performance. In contrast, recent models have not been systematically tested with adversarial-examples in order to show their robustness under severe stress conditions. For that reason, this work evaluates three Transformer-based models (RoBERTa, XLNet, and BERT) in Natural Language Inference (NLI) and Question Answering (QA) tasks to know if they are more robust or if they have the same flaws as their predecessors. As a result, our experiments reveal that RoBERTa, XLNet and BERT are more robust than recurrent neural network models to stress tests for both NLI and QA tasks. Nevertheless, they are still very fragile and demonstrate various unexpected behaviors, thus revealing that there is still room for future improvement in this field.
Anthology ID:
2020.lrec-1.232
Volume:
Proceedings of the 12th Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
1882–1894
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.232
DOI:
Bibkey:
Cite (ACL):
Carlos Aspillaga, Andrés Carvallo, and Vladimir Araujo. 2020. Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1882–1894, Marseille, France. European Language Resources Association.
Cite (Informal):
Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks (Aspillaga et al., LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.232.pdf
Data
MultiNLISQuAD