In this paper, we evaluate an LLM-assisted process for annotating persuasion intent in the context of online manipulation. We focus on the DIDECO taxonomy, a two-tier framework of explicit and implicit manipulation categories based on Speech Act Theory and Cialdini’s influence principles. Our question is not whether such annotations support a downstream classifier, but whether they are reliable enough to be used as labels in the first place. We measure reliability as Cohen’s $\kappa$ between a model and human gold labels, and between models, evaluating GPT-4o, Claude Sonnet 4.5, and Llama 3.3-70B on a synthetic corpus. On DIDECO, models agree more with one another than with human gold, suggesting shared annotation heuristics that diverge from human judgement. On an unlabeled real-world corpus, hierarchical prompting increases inter-model agreement for two of three model pairs, but this result establishes consistency rather than correctness. On a separate human-annotated real-world corpus, we evaluate Claude Sonnet 4.5, Gemma 3-27B, and Qwen 2.5-7B at both top-level and sub-label resolution. Claude and Gemma exceed human agreement at the top level, whereas Claude is the only model to exceed held-out human benchmarks across the full 23-label taxonomy; Gemma also does so when sparse sub-labels are excluded. These results tend to show that LLM annotation reliability depends on the corpus and label granularity, and that inter-model agreement cannot substitute for human validation. We release the prompts, code, and sub-label definitions.

This paper presents DIDECO, the first annotated dataset specifically designed for detecting both explicit and implicit intents in digital communications. We address a critical gap in cybersecurity research by developing a comprehensive taxonomy that distinguishes between explicit communicative goals (what is requested) and implicit persuasion mechanisms (how compliance is engineered). Grounded in Speech Act Theory and persuasion psychology principles, our taxonomy encompasses 20 distinct intent categories across explicit and implicit intents. We annotated 220 LLM-generated spear-phishing emails using a multi-label protocol with six trained annotators, yielding 2,162 intent annotations that reveal the layered complexity of malicious communications. Our analysis demonstrates that sophisticated attacks employ multiple concurrent intents, combining explicit communicative goals with implicit persuasion strategies. This dataset provides resources for developing intent-aware detection systems capable of identifying sophisticated social engineering attacks through semantic analysis.

Cet article propose une taxonomie pour la détection d’intention dans les communications numériques, distinguant les intentions explicites des intentions implicites, basée sur des principes psychologiques de persuasion. Notre approche se distingue par sa capacité à analyser aussi bien les communications numériques légitimes que celles potentiellement malveillantes. Elle repose sur l’identification des intentions sous-jacentes, facilitant ainsi la détection de menaces telles que les arnaques par email (hameçonnage) ou les fraudes sur les réseaux sociaux. Chaque catégorie de la taxonomie est justifiée et illustrée par des exemples de communications correspondant à l’intention associée. Ce travail répond à un manque de ressources dans la recherche sur la détection automatique d’intentions. Il vise à fournir une taxonomie applicable à l’identification des menaces textuelles, notamment les tentatives d’hameçonnage, tout en servant d’outil pédagogique pour sensibiliser le grand public aux stratégies employées dans les communications malveillantes.