Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers

Jakob Smedegaard Andersen, Walid Maalej


Abstract
To maximize the accuracy and increase the overall acceptance of text classifiers, we propose a framework for the efficient, in-operation moderation of classifiers’ output. Our framework focuses on use cases in which F1-scores of modern Neural Networks classifiers (ca. 90%) are still inapplicable in practice. We suggest a semi-automated approach that uses prediction uncertainties to pass unconfident, probably incorrect classifications to human moderators. To minimize the workload, we limit the human moderated data to the point where the accuracy gains saturate and further human effort does not lead to substantial improvements. A series of benchmarking experiments based on three different datasets and three state-of-the-art classifiers show that our framework can improve the classification F1-scores by 5.1 to 11.2% (up to approx. 98 to 99%), while reducing the moderation load up to 73.3% compared to a random moderation.
Anthology ID:
2022.findings-acl.121
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1536–1546
Language:
URL:
https://aclanthology.org/2022.findings-acl.121
DOI:
10.18653/v1/2022.findings-acl.121
Bibkey:
Cite (ACL):
Jakob Smedegaard Andersen and Walid Maalej. 2022. Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1536–1546, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers (Andersen & Maalej, Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.121.pdf
Software:
 2022.findings-acl.121.software.zip
Video:
 https://aclanthology.org/2022.findings-acl.121.mp4
Code
 jsandersen/cmt
Data
IMDb Movie Reviews