Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation

Ian Kivlichan, Zi Lin, Jeremiah Liu, Lucy Vasserman


Abstract
Content moderation is often performed by a collaboration between humans and machine learning models. However, it is not well understood how to design the collaborative process so as to maximize the combined moderator-model system performance. This work presents a rigorous study of this problem, focusing on an approach that incorporates model uncertainty into the collaborative process. First, we introduce principled metrics to describe the performance of the collaborative system under capacity constraints on the human moderator, quantifying how efficiently the combined system utilizes human decisions. Using these metrics, we conduct a large benchmark study evaluating the performance of state-of-the-art uncertainty models under different collaborative review strategies. We find that an uncertainty-based strategy consistently outperforms the widely used strategy based on toxicity scores, and moreover that the choice of review strategy drastically changes the overall system performance. Our results demonstrate the importance of rigorous metrics for understanding and developing effective moderator-model systems for content moderation, as well as the utility of uncertainty estimation in this domain.
Anthology ID:
2021.woah-1.5
Volume:
Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)
Month:
August
Year:
2021
Address:
Online
Editors:
Aida Mostafazadeh Davani, Douwe Kiela, Mathias Lambert, Bertie Vidgen, Vinodkumar Prabhakaran, Zeerak Waseem
Venue:
WOAH
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
36–53
Language:
URL:
https://aclanthology.org/2021.woah-1.5
DOI:
10.18653/v1/2021.woah-1.5
Bibkey:
Cite (ACL):
Ian Kivlichan, Zi Lin, Jeremiah Liu, and Lucy Vasserman. 2021. Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 36–53, Online. Association for Computational Linguistics.
Cite (Informal):
Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation (Kivlichan et al., WOAH 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.woah-1.5.pdf
Video:
 https://aclanthology.org/2021.woah-1.5.mp4
Code
 google/uncertainty-baselines
Data
Civil Comments