In this paper, we evaluate an LLM-assisted process for annotating persuasion intent in the context of online manipulation. We focus on the DIDECO taxonomy, a two-tier framework of explicit and implicit manipulation categories based on Speech Act Theory and Cialdini’s influence principles. Our question is not whether such annotations support a downstream classifier, but whether they are reliable enough to be used as labels in the first place. We measure reliability as Cohen’s $\kappa$ between a model and human gold labels, and between models, evaluating GPT-4o, Claude Sonnet 4.5, and Llama 3.3-70B on a synthetic corpus. On DIDECO, models agree more with one another than with human gold, suggesting shared annotation heuristics that diverge from human judgement. On an unlabeled real-world corpus, hierarchical prompting increases inter-model agreement for two of three model pairs, but this result establishes consistency rather than correctness. On a separate human-annotated real-world corpus, we evaluate Claude Sonnet 4.5, Gemma 3-27B, and Qwen 2.5-7B at both top-level and sub-label resolution. Claude and Gemma exceed human agreement at the top level, whereas Claude is the only model to exceed held-out human benchmarks across the full 23-label taxonomy; Gemma also does so when sparse sub-labels are excluded. These results tend to show that LLM annotation reliability depends on the corpus and label granularity, and that inter-model agreement cannot substitute for human validation. We release the prompts, code, and sub-label definitions.

This paper presents DIDECO, the first annotated dataset specifically designed for detecting both explicit and implicit intents in digital communications. We address a critical gap in cybersecurity research by developing a comprehensive taxonomy that distinguishes between explicit communicative goals (what is requested) and implicit persuasion mechanisms (how compliance is engineered). Grounded in Speech Act Theory and persuasion psychology principles, our taxonomy encompasses 20 distinct intent categories across explicit and implicit intents. We annotated 220 LLM-generated spear-phishing emails using a multi-label protocol with six trained annotators, yielding 2,162 intent annotations that reveal the layered complexity of malicious communications. Our analysis demonstrates that sophisticated attacks employ multiple concurrent intents, combining explicit communicative goals with implicit persuasion strategies. This dataset provides resources for developing intent-aware detection systems capable of identifying sophisticated social engineering attacks through semantic analysis.