<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="penteado-mcgillivray-2026-annotating">
    <titleInfo>
        <title>Annotating Extremist Severity in Brazilian Portuguese: Human–LLM Divergence Across Labels, Evidence, and Interpretation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ricardo</namePart>
        <namePart type="family">Penteado</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Barbara</namePart>
        <namePart type="family">McGillivray</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Agostina</namePart>
            <namePart type="family">Calabrese</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Thomas</namePart>
            <namePart type="family">Davidson</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christine</namePart>
            <namePart type="family">de Kock</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Urja</namePart>
            <namePart type="family">Khurana</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marta</namePart>
            <namePart type="family">Marchiori Manerba</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Paloma</namePart>
            <namePart type="family">Piot</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zeerak</namePart>
            <namePart type="family">Talat</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-487-3</identifier>
    </relatedItem>
    <abstract>This study examines how large language models annotate implicit extremist discourse in Brazilian Portuguese, and where their decisions diverge from human judgement. Using 1,843 risk-enriched segments, we compare three trained human annotators with GPT-4o, Gemini 2.5 Pro, and Claude 4 Sonnet under the same schema-constrained protocol. Each annotation includes a three-level severity label, an extractive evidence span, discourse-pragmatic metadata, and confidence, while human disagreements are adjudicated to produce a single adjudicated reference. We evaluate models against the adjudicated reference using accuracy, macro-F1, and Cohen’s \ensuremathąppa, complement this with a multi-reference evaluation that preserves individual human variation, and additionally examine evidence-span overlap and whether shared evidence leads to the same severity judgement. LLMs agree with one another more than humans do, yet align only moderately with the adjudicated reference. They provide longer and cleaner evidence spans, but overlap with human rationales does not predict label correctness. Errors concentrate in the implicit class, where models often recognise similar cues but assign different severity levels. These findings identify agreement, confidence, and rationale overlap as unreliable proxies for annotation validity, and motivate evidence- and interpretation-level diagnostics for harmful-content annotation.</abstract>
    <identifier type="citekey">penteado-mcgillivray-2026-annotating</identifier>
    <location>
        <url>https://aclanthology.org/2026.woah-1.25/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>284</start>
            <end>297</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Annotating Extremist Severity in Brazilian Portuguese: Human–LLM Divergence Across Labels, Evidence, and Interpretation
%A Penteado, Ricardo
%A McGillivray, Barbara
%Y Calabrese, Agostina
%Y Davidson, Thomas
%Y de Kock, Christine
%Y Khurana, Urja
%Y Marchiori Manerba, Marta
%Y Piot, Paloma
%Y Talat, Zeerak
%S Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-487-3
%F penteado-mcgillivray-2026-annotating
%X This study examines how large language models annotate implicit extremist discourse in Brazilian Portuguese, and where their decisions diverge from human judgement. Using 1,843 risk-enriched segments, we compare three trained human annotators with GPT-4o, Gemini 2.5 Pro, and Claude 4 Sonnet under the same schema-constrained protocol. Each annotation includes a three-level severity label, an extractive evidence span, discourse-pragmatic metadata, and confidence, while human disagreements are adjudicated to produce a single adjudicated reference. We evaluate models against the adjudicated reference using accuracy, macro-F1, and Cohen’s \ensuremathąppa, complement this with a multi-reference evaluation that preserves individual human variation, and additionally examine evidence-span overlap and whether shared evidence leads to the same severity judgement. LLMs agree with one another more than humans do, yet align only moderately with the adjudicated reference. They provide longer and cleaner evidence spans, but overlap with human rationales does not predict label correctness. Errors concentrate in the implicit class, where models often recognise similar cues but assign different severity levels. These findings identify agreement, confidence, and rationale overlap as unreliable proxies for annotation validity, and motivate evidence- and interpretation-level diagnostics for harmful-content annotation.
%U https://aclanthology.org/2026.woah-1.25/
%P 284-297

Markdown (Informal)

[Annotating Extremist Severity in Brazilian Portuguese: Human–LLM Divergence Across Labels, Evidence, and Interpretation](https://aclanthology.org/2026.woah-1.25/) (Penteado & McGillivray, WOAH 2026)

• Annotating Extremist Severity in Brazilian Portuguese: Human–LLM Divergence Across Labels, Evidence, and Interpretation (Penteado & McGillivray, WOAH 2026)

ACL

• Ricardo Penteado and Barbara McGillivray. 2026. Annotating Extremist Severity in Brazilian Portuguese: Human–LLM Divergence Across Labels, Evidence, and Interpretation. In Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH), pages 284–297, Budapest, Hungary. Association for Computational Linguistics.