From Toxicity in Online Comments to Incivility in American News: Proceed with Caution

Anushree Hede, Oshin Agarwal, Linda Lu, Diana C. Mutz, Ani Nenkova


Abstract
The ability to quantify incivility online, in news and in congressional debates is of great interest to political scientists. Computational tools for detecting online incivility for English are now fairly accessible and potentially could be applied more broadly. We test the Jigsaw Perspective API for its ability to detect the degree of incivility on a corpus that we developed, consisting of manual annotations of civility in American news. We demonstrate that toxicity models, as exemplified by Perspective, are inadequate for the analysis of incivility in news. We carry out error analysis that points to the need to develop methods to remove spurious correlations between words often mentioned in the news, especially identity descriptors and incivility. Without such improvements, applying Perspective or similar models on news is likely to lead to wrong conclusions, that are not aligned with the human perception of incivility.
Anthology ID:
2021.eacl-main.225
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Editors:
Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2620–2630
Language:
URL:
https://aclanthology.org/2021.eacl-main.225
DOI:
10.18653/v1/2021.eacl-main.225
Bibkey:
Cite (ACL):
Anushree Hede, Oshin Agarwal, Linda Lu, Diana C. Mutz, and Ani Nenkova. 2021. From Toxicity in Online Comments to Incivility in American News: Proceed with Caution. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2620–2630, Online. Association for Computational Linguistics.
Cite (Informal):
From Toxicity in Online Comments to Incivility in American News: Proceed with Caution (Hede et al., EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.225.pdf