RG PA at SemEval-2021 Task 1: A Contextual Attention-based Model with RoBERTa for Lexical Complexity Prediction

Gang Rao, Maochang Li, Xiaolong Hou, Lianxin Jiang, Yang Mo, Jianping Shen


Abstract
In this paper we propose a contextual attention based model with two-stage fine-tune training using RoBERTa. First, we perform the first-stage fine-tune on corpus with RoBERTa, so that the model can learn some prior domain knowledge. Then we get the contextual embedding of context words based on the token-level embedding with the fine-tuned model. And we use Kfold cross-validation to get K models and ensemble them to get the final result. Finally, we attain the 2nd place in the final evaluation phase of sub-task 2 with pearson correlation of 0.8575.
Anthology ID:
2021.semeval-1.79
Volume:
Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP | SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
623–626
Language:
URL:
https://aclanthology.org/2021.semeval-1.79
DOI:
10.18653/v1/2021.semeval-1.79
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.semeval-1.79.pdf