Online abuse research relies on a substantial measurement infrastructure comprising datasets, shared tasks, diagnostic suites, taxonomies, and evaluation practices that shape which harms become visible, comparable, and citable. We introduce Dataset Archaeology as a method for auditing that infrastructure and apply it to the first published decade of Abusive Language Online and Workshop on Online Abuse and Harms (ALW/WOAH) research. Our audit combines a reconciled corpus of 203 archival ALW/WOAH papers from 2017-2025 with a 60-resource registry of online-harm datasets and diagnostic benchmarks. The field has expanded beyond early hate-speech, offensive-language, and toxicity benchmarks toward diagnostic, perspectivist, multilingual, and multimodal resources. At the same time, much of the empirical record continues to be shaped by a narrower set of recurring design choices: short-form text, English, Twitter/X, coarse labels, thin context, and limited documentation of labeling authority. Across the dataset and diagnostic resources, documentation of annotation authority, annotator perspective, and disagreement remains uneven. Dataset influence also changes when mentions are separated from central use. Among 404 detected paper-dataset matches for the top reused resources, only 52 were verified as central experimental or analytical uses. We argue that progress in online-harm NLP should be evaluated as progress in measurement instruments, not only model performance, and we offer a next-decade agenda for contextual, cross-cultural, exposure-aware, and stakeholder-legitimate harm measurement.

Socially motivated NLP research often connects technical contributions to intended benefits for people, institutions, or communities. The outcome invoked in a paper’s motivation, however, is not always the outcome evaluated in the study. This makes it difficult to determine what a result establishes, which transitions in the impact pathway remain assumptions, and how evidence should accumulate across technical, user, operational, and social studies. We introduce SOLIAS, Social Outcomes for Language-Integrated AI Systems, a reporting standard for social-impact claims in NLP. SOLIAS has three linked components: six evidence levels that identify the highest outcome layer directly evaluated, a claim-evidence alignment measure that relates conclusions to demonstrated outcomes, and a 20-item checklist for reporting populations, contexts, mechanisms, evaluation, safeguards, maintenance, and claim scope. In a retrospective audit of a multi-domain corpus of 168 research papers, demonstrated evidence is concentrated at the technical layer: 97.6% of papers are classified at Level 0, while 35.7% have a claim–evidence gap of at least two outcome layers. The audit illustrates how SOLIAS distinguishes valuable technical work from evidence about downstream outcomes without treating either as a substitute for the other. We release author and reviewer templates, operational definitions, worked examples, and a machine-readable schema to support adoption. SOLIAS is intended to make the scope of evidence explicit and to support research in which technical, user, institutional, and social outcomes are evaluated cumulatively.

LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators. These pipelines typically assume that judgments are stable properties of fixed inputs. We show that this assumption does not hold under interaction.We study post-decision manipulability: the extent to which an evaluation outcome can be altered through subsequent conversation with the judge after an initial decision has been made. Across controlled experiments on MT-Bench and AlpacaEval, we find that LLM judges are highly stable under repeated and neutral reevaluation, yet become substantially reversible under targeted post-decision challenge. An anti-baseline challenge protocol shows that stable judgments can be overturned through motivated interaction, while a counterbalanced target-validation protocol separates this reversibility from net target-directed steering.These reversals have practical consequences: they can degrade agreement with human preferences, shift benchmark rankings, and produce harmful evaluation changes despite high self-reported confidence. Authority framing is especially destabilizing, and revised judgments are often accompanied by low-overlap justifications, suggesting post hoc rationalization rather than reliable error correction. We introduce the Evaluation Robustness Score (ERS) to quantify interactional robustness by combining reversal susceptibility with counterbalanced directional effects. Our findings identify post-decision interaction as a distinct failure mode for LLM-as-judge evaluation and motivate evaluation protocols that measure not only static agreement, but robustness under challenge.