Toxicity to Harm: Dataset Archaeology of a Decade of Online Abuse Research

Srimonti Dutta, Akshata Moharir


Abstract
Online abuse research relies on a substantial measurement infrastructure comprising datasets, shared tasks, diagnostic suites, taxonomies, and evaluation practices that shape which harms become visible, comparable, and citable. We introduce Dataset Archaeology as a method for auditing that infrastructure and apply it to the first published decade of Abusive Language Online and Workshop on Online Abuse and Harms (ALW/WOAH) research. Our audit combines a reconciled corpus of 203 archival ALW/WOAH papers from 2017-2025 with a 60-resource registry of online-harm datasets and diagnostic benchmarks. The field has expanded beyond early hate-speech, offensive-language, and toxicity benchmarks toward diagnostic, perspectivist, multilingual, and multimodal resources. At the same time, much of the empirical record continues to be shaped by a narrower set of recurring design choices: short-form text, English, Twitter/X, coarse labels, thin context, and limited documentation of labeling authority. Across the dataset and diagnostic resources, documentation of annotation authority, annotator perspective, and disagreement remains uneven. Dataset influence also changes when mentions are separated from central use. Among 404 detected paper-dataset matches for the top reused resources, only 52 were verified as central experimental or analytical uses. We argue that progress in online-harm NLP should be evaluated as progress in measurement instruments, not only model performance, and we offer a next-decade agenda for contextual, cross-cultural, exposure-aware, and stakeholder-legitimate harm measurement.
Anthology ID:
2026.woah-1.37
Volume:
Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH)
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Agostina Calabrese, Thomas Davidson, Christine de Kock, Urja Khurana, Marta Marchiori Manerba, Paloma Piot, Zeerak Talat
Venues:
WOAH | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
450–465
Language:
URL:
https://aclanthology.org/2026.woah-1.37/
DOI:
Bibkey:
Cite (ACL):
Srimonti Dutta and Akshata Moharir. 2026. Toxicity to Harm: Dataset Archaeology of a Decade of Online Abuse Research. In Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH), pages 450–465, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Toxicity to Harm: Dataset Archaeology of a Decade of Online Abuse Research (Dutta & Moharir, WOAH 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.woah-1.37.pdf