Ricardo Penteado

Author directory

2026

This study examines how large language models annotate implicit extremist discourse in Brazilian Portuguese, and where their decisions diverge from human judgement. Using 1,843 risk-enriched segments, we compare three trained human annotators with GPT-4o, Gemini 2.5 Pro, and Claude 4 Sonnet under the same schema-constrained protocol. Each annotation includes a three-level severity label, an extractive evidence span, discourse-pragmatic metadata, and confidence, while human disagreements are adjudicated to produce a single adjudicated reference. We evaluate models against the adjudicated reference using accuracy, macro-F1, and Cohen’s κ, complement this with a multi-reference evaluation that preserves individual human variation, and additionally examine evidence-span overlap and whether shared evidence leads to the same severity judgement. LLMs agree with one another more than humans do, yet align only moderately with the adjudicated reference. They provide longer and cleaner evidence spans, but overlap with human rationales does not predict label correctness. Errors concentrate in the implicit class, where models often recognise similar cues but assign different severity levels. These findings identify agreement, confidence, and rationale overlap as unreliable proxies for annotation validity, and motivate evidence- and interpretation-level diagnostics for harmful-content annotation.