SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge

Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, Minlie Huang


Abstract
Most of the existing pre-trained language representation models neglect to consider the linguistic knowledge of texts, which can promote language understanding in NLP tasks. To benefit the downstream tasks in sentiment analysis, we propose a novel language representation model called SentiLARE, which introduces word-level linguistic knowledge including part-of-speech tag and sentiment polarity (inferred from SentiWordNet) into pre-trained models. We first propose a context-aware sentiment attention mechanism to acquire the sentiment polarity of each word with its part-of-speech tag by querying SentiWordNet. Then, we devise a new pre-training task called label-aware masked language model to construct knowledge-aware language representation. Experiments show that SentiLARE obtains new state-of-the-art performance on a variety of sentiment analysis tasks.
Anthology ID:
2020.emnlp-main.567
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6975–6988
Language:
URL:
https://aclanthology.org/2020.emnlp-main.567
DOI:
10.18653/v1/2020.emnlp-main.567
Bibkey:
Cite (ACL):
Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2020. SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6975–6988, Online. Association for Computational Linguistics.
Cite (Informal):
SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge (Ke et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.567.pdf
Video:
 https://slideslive.com/38938931
Code
 thu-coai/SentiLARE
Data
IMDb Movie ReviewsSICKSST