@inproceedings{patel-2026-fifty,
title = "Fifty Tokens of Signal: A Cross-Dataset Audit of Overlap, Shortcuts, and Label Divergence in Online-Abuse Benchmarks",
author = "Patel, Dhruv",
editor = "Calabrese, Agostina and
Davidson, Thomas and
de Kock, Christine and
Khurana, Urja and
Marchiori Manerba, Marta and
Piot, Paloma and
Talat, Zeerak",
booktitle = "Proceedings of the 10th Workshop on Online Abuse and Harms ({WOAH})",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.woah-1.27/",
pages = "307--321",
ISBN = "979-8-89176-487-3",
abstract = "A decade after the first Workshop on Online Abuse and Harms, abuse detection is evaluated on a sprawling collection of public datasets that are usually treated as independent and interchangeable. We audit five widely used English benchmarks (Davidson, DynaHate, HateXplain, TweetEval-Hate, and the Stormfront corpus; 109,698 posts) and, by measuring contamination, lexical shortcuts, label conflict, and transfer on one common scale, argue a single thesis: for these benchmarks the evidence points to a conceptual measurement bottleneck (what counts as abuse) more than to data hygiene (contamination or leakage), a reading we hold with the caution that the strict-cutoff conflict sample is small. Three findings carry this claim. (i) The data-hygiene worries are small here: cross-dataset duplication is negligible (only 17 near-duplicate pairs at Jaccard = 0.8 across 108,946 unique texts, 5 of them exact), and train/test near-duplicate leakage, though common (up to 9.8{\%} of test items), barely changes macro-F1 for the pipeline we study, both for TF-IDF logistic regression and for a fine-tuned distilBERT on the two official-split corpora. (ii) The signal these benchmarks reward is concentrated in a few tokens: a TF-IDF model restricted to 50 tokens retains roughly 89 to 95{\%} of full-model macro-F1 on every dataset, and, against the natural objection that this is a bag-of-words artifact, a fine-tuned distilBERT trained on only those 50 tokens per post still retains 81 to 92{\%} across all five datasets (lowest on the adversarially debiased DynaHate). (iii) Where the same surface text recurs across corpora, its binary labels frequently flip, isolating definition from sampling: at a loose Jaccard = 0.5 near-duplicate cutoff the clean per-item conflict rate is 48.6{\%} (103 of 212 shared texts, though many of these loose matches share a profanity skeleton rather than identical text); at the strict same-text Jaccard = 0.8 cutoff the clean shared set is small (n=18, 10 in conflict) and the disagreement is a Davidson-driven offensive-versus-hate split. Transfer is correspondingly poor (a 0.20 macro-F1 gap) but, given negligible contamination, reflects definitional and distributional divergence rather than leakage; we cannot separate those two, so the conceptual claim rests on the item-level comparison rather than on the transfer gap. Holding the near-duplicate graph fixed and sending both corpora{'}s offensive class to not-abuse moves the strict-cutoff conflict rate from 55.6{\%} to 22.2{\%}, so much of that disagreement is the offensive-versus-hate boundary rather than disagreement about hate itself. We release abuse-audit, a one-command tool that emits the same per-dataset report card for any new corpus, and recommend reporting a shortcut score and a cross-dataset transfer cell as standard practice."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="patel-2026-fifty">
<titleInfo>
<title>Fifty Tokens of Signal: A Cross-Dataset Audit of Overlap, Shortcuts, and Label Divergence in Online-Abuse Benchmarks</title>
</titleInfo>
<name type="personal">
<namePart type="given">Dhruv</namePart>
<namePart type="family">Patel</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH)</title>
</titleInfo>
<name type="personal">
<namePart type="given">Agostina</namePart>
<namePart type="family">Calabrese</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Thomas</namePart>
<namePart type="family">Davidson</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Christine</namePart>
<namePart type="family">de Kock</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Urja</namePart>
<namePart type="family">Khurana</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Marta</namePart>
<namePart type="family">Marchiori Manerba</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Paloma</namePart>
<namePart type="family">Piot</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zeerak</namePart>
<namePart type="family">Talat</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-487-3</identifier>
</relatedItem>
<abstract>A decade after the first Workshop on Online Abuse and Harms, abuse detection is evaluated on a sprawling collection of public datasets that are usually treated as independent and interchangeable. We audit five widely used English benchmarks (Davidson, DynaHate, HateXplain, TweetEval-Hate, and the Stormfront corpus; 109,698 posts) and, by measuring contamination, lexical shortcuts, label conflict, and transfer on one common scale, argue a single thesis: for these benchmarks the evidence points to a conceptual measurement bottleneck (what counts as abuse) more than to data hygiene (contamination or leakage), a reading we hold with the caution that the strict-cutoff conflict sample is small. Three findings carry this claim. (i) The data-hygiene worries are small here: cross-dataset duplication is negligible (only 17 near-duplicate pairs at Jaccard = 0.8 across 108,946 unique texts, 5 of them exact), and train/test near-duplicate leakage, though common (up to 9.8% of test items), barely changes macro-F1 for the pipeline we study, both for TF-IDF logistic regression and for a fine-tuned distilBERT on the two official-split corpora. (ii) The signal these benchmarks reward is concentrated in a few tokens: a TF-IDF model restricted to 50 tokens retains roughly 89 to 95% of full-model macro-F1 on every dataset, and, against the natural objection that this is a bag-of-words artifact, a fine-tuned distilBERT trained on only those 50 tokens per post still retains 81 to 92% across all five datasets (lowest on the adversarially debiased DynaHate). (iii) Where the same surface text recurs across corpora, its binary labels frequently flip, isolating definition from sampling: at a loose Jaccard = 0.5 near-duplicate cutoff the clean per-item conflict rate is 48.6% (103 of 212 shared texts, though many of these loose matches share a profanity skeleton rather than identical text); at the strict same-text Jaccard = 0.8 cutoff the clean shared set is small (n=18, 10 in conflict) and the disagreement is a Davidson-driven offensive-versus-hate split. Transfer is correspondingly poor (a 0.20 macro-F1 gap) but, given negligible contamination, reflects definitional and distributional divergence rather than leakage; we cannot separate those two, so the conceptual claim rests on the item-level comparison rather than on the transfer gap. Holding the near-duplicate graph fixed and sending both corpora’s offensive class to not-abuse moves the strict-cutoff conflict rate from 55.6% to 22.2%, so much of that disagreement is the offensive-versus-hate boundary rather than disagreement about hate itself. We release abuse-audit, a one-command tool that emits the same per-dataset report card for any new corpus, and recommend reporting a shortcut score and a cross-dataset transfer cell as standard practice.</abstract>
<identifier type="citekey">patel-2026-fifty</identifier>
<location>
<url>https://aclanthology.org/2026.woah-1.27/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>307</start>
<end>321</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Fifty Tokens of Signal: A Cross-Dataset Audit of Overlap, Shortcuts, and Label Divergence in Online-Abuse Benchmarks
%A Patel, Dhruv
%Y Calabrese, Agostina
%Y Davidson, Thomas
%Y de Kock, Christine
%Y Khurana, Urja
%Y Marchiori Manerba, Marta
%Y Piot, Paloma
%Y Talat, Zeerak
%S Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-487-3
%F patel-2026-fifty
%X A decade after the first Workshop on Online Abuse and Harms, abuse detection is evaluated on a sprawling collection of public datasets that are usually treated as independent and interchangeable. We audit five widely used English benchmarks (Davidson, DynaHate, HateXplain, TweetEval-Hate, and the Stormfront corpus; 109,698 posts) and, by measuring contamination, lexical shortcuts, label conflict, and transfer on one common scale, argue a single thesis: for these benchmarks the evidence points to a conceptual measurement bottleneck (what counts as abuse) more than to data hygiene (contamination or leakage), a reading we hold with the caution that the strict-cutoff conflict sample is small. Three findings carry this claim. (i) The data-hygiene worries are small here: cross-dataset duplication is negligible (only 17 near-duplicate pairs at Jaccard = 0.8 across 108,946 unique texts, 5 of them exact), and train/test near-duplicate leakage, though common (up to 9.8% of test items), barely changes macro-F1 for the pipeline we study, both for TF-IDF logistic regression and for a fine-tuned distilBERT on the two official-split corpora. (ii) The signal these benchmarks reward is concentrated in a few tokens: a TF-IDF model restricted to 50 tokens retains roughly 89 to 95% of full-model macro-F1 on every dataset, and, against the natural objection that this is a bag-of-words artifact, a fine-tuned distilBERT trained on only those 50 tokens per post still retains 81 to 92% across all five datasets (lowest on the adversarially debiased DynaHate). (iii) Where the same surface text recurs across corpora, its binary labels frequently flip, isolating definition from sampling: at a loose Jaccard = 0.5 near-duplicate cutoff the clean per-item conflict rate is 48.6% (103 of 212 shared texts, though many of these loose matches share a profanity skeleton rather than identical text); at the strict same-text Jaccard = 0.8 cutoff the clean shared set is small (n=18, 10 in conflict) and the disagreement is a Davidson-driven offensive-versus-hate split. Transfer is correspondingly poor (a 0.20 macro-F1 gap) but, given negligible contamination, reflects definitional and distributional divergence rather than leakage; we cannot separate those two, so the conceptual claim rests on the item-level comparison rather than on the transfer gap. Holding the near-duplicate graph fixed and sending both corpora’s offensive class to not-abuse moves the strict-cutoff conflict rate from 55.6% to 22.2%, so much of that disagreement is the offensive-versus-hate boundary rather than disagreement about hate itself. We release abuse-audit, a one-command tool that emits the same per-dataset report card for any new corpus, and recommend reporting a shortcut score and a cross-dataset transfer cell as standard practice.
%U https://aclanthology.org/2026.woah-1.27/
%P 307-321
Markdown (Informal)
[Fifty Tokens of Signal: A Cross-Dataset Audit of Overlap, Shortcuts, and Label Divergence in Online-Abuse Benchmarks](https://aclanthology.org/2026.woah-1.27/) (Patel, WOAH 2026)
ACL