Knowledge Enhanced Masked Language Model for Stance Detection

Kornraphop Kawintiranon, Lisa Singh


Abstract
Detecting stance on Twitter is especially challenging because of the short length of each tweet, the continuous coinage of new terminology and hashtags, and the deviation of sentence structure from standard prose. Fine-tuned language models using large-scale in-domain data have been shown to be the new state-of-the-art for many NLP tasks, including stance detection. In this paper, we propose a novel BERT-based fine-tuning method that enhances the masked language model for stance detection. Instead of random token masking, we propose using a weighted log-odds-ratio to identify words with high stance distinguishability and then model an attention mechanism that focuses on these words. We show that our proposed approach outperforms the state of the art for stance detection on Twitter data about the 2020 US Presidential election.
Anthology ID:
2021.naacl-main.376
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4725–4735
Language:
URL:
https://aclanthology.org/2021.naacl-main.376
DOI:
10.18653/v1/2021.naacl-main.376
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.376.pdf