We present the first survey study of German-language counterspeech in an Austrian context, where participants ranked LLM-generated responses to offensive comments involving xenophobia, sexual objectification, and political incivility. We compare generations from five state-of-the-art LLMs and additionally examine whether participants perceive them as human- or machine-written. Our results show substantial disagreement in rankings, indicating that counterspeech preferences are highly subjective. Although participants express a clear preference for human-written counterspeech in real-world settings, they fail to reliably identify LLM-generated responses as machine-written. These findings point to the need for personalized counterspeech systems that account for users’ diverse preferences, rather than assuming a single universally effective response. Such systems could combine LLMs’ capacity to generate high-quality counterspeech at scale with human involvement.

Comparative analyses of NLP research, such as identifying who is shaping a subfield, how a topic is shifting geographically, where method- ological gaps lie between research communi- ties, require institutional and topical metadata at a density and granularity the ACL Anthol- ogy does not itself provide. Large-scale re- sources offer partial support: OpenAlex pro- vides high-quality institutional metadata where it can match ACL papers (32% of our gold set), but the DOI dependency limits its cov- erage on older Anthology publications; and the S2ORC bulk release contains 52,013 ACL Anthology fulltexts but populates structured author–affiliation annotations on a small fraction of them in its current state. We present ANTHEM (ANTHology with Enriched Meta- data), a complementary resource that narrows this gap for NLP publications. ANTHEM links 117,386 ACL Anthology papers to Research Or- ganization Registry (ROR)-normalized author affiliations and Computer Science Ontology (CSO) topic labels via a multi-parser, LLM- based extraction pipeline that retains evidence spans for every decision, achieving F1hier of 0.760 on a 781-paper decade-stratified gold set, where parent–child ROR identifiers are treated as equivalent (e.g., a department and its parent university). On top of the corpus we release a benchmark of 99 structured comparison ques- tions across five categories, evaluated under a programmatic gold-set construction in which ANTHEM’s own metadata defines the evalu- ation ground truth. All data, code, and infras- tructure are released under CC-BY-SA-NC.

Toxicity labels at sub-document granularity and disaggregated labels lead to more nuanced and personalized toxicity classification and facilitate analysis. We re-annotate a subset of 1983 posts of the Jigsaw Toxic Comment Classification Challenge and provide disaggregated toxicity labels and spans that identify inappropriate language and targets of toxic statements. Manual analysis shows that five annotations per instance effectively capture meaningful disagreement patterns and allow for finer distinctions between genuine disagreement and that arising from annotation error or inconsistency. Our main findings are: (1) Disagreement often stems from divergent interpretations of edge-case toxicity (2) Disagreement is especially high in cases of toxic statements involving non-human targets (3) Disagreement on whether a passage consists of inappropriate language occurs not only on inherently questionable terms, but also on words that may be inappropriate in specific contexts while remaining acceptable in others (4) Transformer-based models effectively learn from aggregated data that reduces false negative classifications by being more sensitive towards minority opinions for posts to be toxic. We publish the new annotations under the CC BY 4.0 license.

Model interpretability in toxicity detection greatly profits from token-level annotations. However, currently, such annotations are only available in English. We introduce a dataset annotated for offensive language detection sourced from a news forum, notable for its incorporation of the Austrian German dialect, comprising 4,562 user comments. In addition to binary offensiveness classification, we identify spans within each comment constituting vulgar language or representing targets of offensive statements. We evaluate fine-tuned Transformer models as well as large language models in a zero- and few-shot fashion. The results indicate that while fine-tuned models excel in detecting linguistic peculiarities such as vulgar dialect, large language models demonstrate superior performance in detecting offensiveness in AustroTox.

The definitions of abusive, offensive, toxic and uncivil comments used for annotating corpora for automated content moderation are highly intersected and researchers call for their disambiguation. We summarize the definitions of these terms as they appear in 23 papers across different fields. We compare examples given for uncivil, offensive, and toxic comments, attempting to foster more unified scientific resources. Additionally, we stress that the term incivility that frequently appears in social science literature has hardly been mentioned in the literature we analyzed that focuses on computational linguistics and natural language processing.