JBNU-CCLab at SemEval-2022 Task 7: DeBERTa for Identifying Plausible Clarifications in Instructional Texts

Daewook Kang, Sung-Min Lee, Eunhwan Park, Seung-Hoon Na


Abstract
In this study, we examine the ability of contextualized representations of pretrained language model to distinguish whether sequences from instructional articles are plausible or implausible. Towards this end, we compare the BERT, RoBERTa, and DeBERTa models using simple classifiers based on the sentence representations of the [CLS] tokens and perform a detailed analysis by visualizing the representations of the [CLS] tokens of the models. In the experimental results of Subtask A: Multi-Class Classification, DeBERTa exhibits the best performance and produces a more distinguishable representation across different labels. Submitting an ensemble of 10 DeBERTa-based models, our final system achieves an accuracy of 61.4% and is ranked fifth out of models submitted by eight teams. Further in-depth results suggest that the abilities of pretrained language models for the plausibility detection task are more strongly affected by their model structures or attention designs than by their model sizes.
Anthology ID:
2022.semeval-1.147
Volume:
Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, Shyam Ratan
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
1050–1055
Language:
URL:
https://aclanthology.org/2022.semeval-1.147
DOI:
10.18653/v1/2022.semeval-1.147
Bibkey:
Cite (ACL):
Daewook Kang, Sung-Min Lee, Eunhwan Park, and Seung-Hoon Na. 2022. JBNU-CCLab at SemEval-2022 Task 7: DeBERTa for Identifying Plausible Clarifications in Instructional Texts. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1050–1055, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
JBNU-CCLab at SemEval-2022 Task 7: DeBERTa for Identifying Plausible Clarifications in Instructional Texts (Kang et al., SemEval 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.semeval-1.147.pdf
Video:
 https://aclanthology.org/2022.semeval-1.147.mp4