Syntactical Analysis of the Weaknesses of Sentiment Analyzers

Rohil Verma, Samuel Kim, David Walter


Abstract
We carry out a syntactic analysis of two state-of-the-art sentiment analyzers, Google Cloud Natural Language and Stanford CoreNLP, to assess their classification accuracy on sentences with negative polarity items. We were motivated by the absence of studies investigating sentiment analyzer performance on sentences with polarity items, a common construct in human language. Our analysis focuses on two sentential structures: downward entailment and non-monotone quantifiers; and demonstrates weaknesses of Google Natural Language and CoreNLP in capturing polarity item information. We describe the particular syntactic phenomenon that these analyzers fail to understand that any ideal sentiment analyzer must. We also provide a set of 150 test sentences that any ideal sentiment analyzer must be able to understand.
Anthology ID:
D18-1141
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Editors:
Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1122–1127
Language:
URL:
https://aclanthology.org/D18-1141
DOI:
10.18653/v1/D18-1141
Bibkey:
Cite (ACL):
Rohil Verma, Samuel Kim, and David Walter. 2018. Syntactical Analysis of the Weaknesses of Sentiment Analyzers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1122–1127, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Syntactical Analysis of the Weaknesses of Sentiment Analyzers (Verma et al., EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1141.pdf
Attachment:
 D18-1141.Attachment.zip