Nicolas Loaiza

Author directory

2026

Hate speech detection is inherently subjective because annotators routinely disagree on the same item, and collapsing their labels into a majority vote discards information about how contested the item actually is. Large language models exhibit variation by returning different labels for the same input across random seeds. We ask whether this variation empirically lines up with human disagreement. We operationalize annotator disagreement via entropy and the LLM instability via flip rate and model entropy across five seeds at fixed temperature, then measure their rank correlation on the full CREHate dataset using three models from both open- and closed-source families. The alignment with human disagreement is model dependent: Gemini 2.5 Flash and Llama 3.1 8B show significant positive correlations, while Mistral 7B shows only a marginal effect. The relationship is also modest in absolute scale: most items cluster at a flip rate of zero, and the signal is concentrated in a small fraction of high-disagreement items that also produce model instability. Qualitative analysis shows that culturally contested items, where human review is needed, sometimes produce zero model instability, as models classify them in opposite directions across cultural contexts. Our findings do not show evidence that seed-based instability is a reliable proxy for human disagreement for all LLMs.