TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training

Thomas Müller, Julian Eisenschlos, Syrine Krichene


Abstract
We present the TAPAS contribution to the Shared Task on Statement Verification and Evidence Finding with Tables (SemEval 2021 Task 9, Wang et al. (2021)). SEM TAB FACT Task A is a classification task of recognizing if a statement is entailed, neutral or refuted by the content of a given table. We adopt the binary TAPAS model of Eisenschlos et al. (2020) to this task. We learn two binary classification models: A first model to predict if a statement is neutral or non-neutral and a second one to predict if it is entailed or refuted. As the shared task training set contains only entailed or refuted examples, we generate artificial neutral examples to train the first model. Both models are pre-trained using a MASKLM objective, intermediate counter-factual and synthetic data (Eisenschlos et al., 2020) and TABFACT (Chen et al., 2020), a large table entailment dataset. We find that the artificial neutral examples are somewhat effective at training the first model, achieving 68.03 test F1 versus the 60.47 of a majority baseline. For the second stage, we find that the pre-training on the intermediate data and TABFACT improves the results over MASKLM pre-training (68.03 vs 57.01).
Anthology ID:
2021.semeval-1.51
Volume:
Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)
Month:
August
Year:
2021
Address:
Online
Editors:
Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, Xiaodan Zhu
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
423–430
Language:
URL:
https://aclanthology.org/2021.semeval-1.51
DOI:
10.18653/v1/2021.semeval-1.51
Bibkey:
Cite (ACL):
Thomas Müller, Julian Eisenschlos, and Syrine Krichene. 2021. TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 423–430, Online. Association for Computational Linguistics.
Cite (Informal):
TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training (Müller et al., SemEval 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.semeval-1.51.pdf
Data
SQATabFact