Agree or Disagree: Predicting Judgments on Nuanced Assertions

Michael Wojatzki, Torsten Zesch, Saif Mohammad, Svetlana Kiritchenko


Abstract
Being able to predict whether people agree or disagree with an assertion (i.e. an explicit, self-contained statement) has several applications ranging from predicting how many people will like or dislike a social media post to classifying posts based on whether they are in accordance with a particular point of view. We formalize this as two NLP tasks: predicting judgments of (i) individuals and (ii) groups based on the text of the assertion and previous judgments. We evaluate a wide range of approaches on a crowdsourced data set containing over 100,000 judgments on over 2,000 assertions. We find that predicting individual judgments is a hard task with our best results only slightly exceeding a majority baseline, but that judgments of groups can be more reliably predicted using a Siamese neural network, which outperforms all other approaches by a wide margin.
Anthology ID:
S18-2026
Volume:
Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Venue:
SemEval
SIGs:
SIGSEM | SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
214–224
Language:
URL:
https://aclanthology.org/S18-2026
DOI:
10.18653/v1/S18-2026
Bibkey:
Cite (ACL):
Michael Wojatzki, Torsten Zesch, Saif Mohammad, and Svetlana Kiritchenko. 2018. Agree or Disagree: Predicting Judgments on Nuanced Assertions. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 214–224, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Agree or Disagree: Predicting Judgments on Nuanced Assertions (Wojatzki et al., SemEval 2018)
Copy Citation:
PDF:
https://aclanthology.org/S18-2026.pdf
Code
 muchafel/judgmentPrediction