‘Am I the Bad One’? Predicting the Moral Judgement of the Crowd Using Pre–trained Language Models

Areej Alhassan, Jinkai Zhang, Viktor Schlegel


Abstract
Natural language processing (NLP) has been shown to perform well in various tasks, such as answering questions, ascertaining natural language inference and anomaly detection. However, there are few NLP-related studies that touch upon the moral context conveyed in text. This paper studies whether state-of-the-art, pre-trained language models are capable of passing moral judgments on posts retrieved from a popular Reddit user board. Reddit is a social discussion website and forum where posts are promoted by users through a voting system. In this work, we construct a dataset that can be used for moral judgement tasks by collecting data from the AITA? (Am I the A*******?) subreddit. To model our task, we harnessed the power of pre-trained language models, including BERT, RoBERTa, RoBERTa-large, ALBERT and Longformer. We then fine-tuned these models and evaluated their ability to predict the correct verdict as judged by users for each post in the datasets. RoBERTa showed relative improvements across the three datasets, exhibiting a rate of 87% accuracy and a Matthews correlation coefficient (MCC) of 0.76, while the use of the Longformer model slightly improved the performance when used with longer sequences, achieving 87% accuracy and 0.77 MCC.
Anthology ID:
2022.lrec-1.28
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
267–276
Language:
URL:
https://aclanthology.org/2022.lrec-1.28
DOI:
Bibkey:
Cite (ACL):
Areej Alhassan, Jinkai Zhang, and Viktor Schlegel. 2022. ‘Am I the Bad One’? Predicting the Moral Judgement of the Crowd Using Pre–trained Language Models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 267–276, Marseille, France. European Language Resources Association.
Cite (Informal):
‘Am I the Bad One’? Predicting the Moral Judgement of the Crowd Using Pre–trained Language Models (Alhassan et al., LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.28.pdf