Andrew Paverd
Author directory2026
LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge
Sahar Abdelnabi | Aideen Fay | Ahmed Salem | Egor Zverev | Kai-Chieh Liao | Chi-Huang Liu | Chun Chih Kuo | Jannis Weigend | Danyael Manlangit | Alex Apostolov | Haris Umair | João Donato | Masayuki Kawakita | Athar Mahboob | Tran Huu Bach | Tsun-Han Chiang | Myeongjin Cho | Hajin Choi | Byeonghyeon Kim | Hyeonjin Lee | Benjamin Pannell | Conor McCauley | Mark Russinovich | Andrew Paverd | Giovanni Cherubin
Findings of the Association for Computational Linguistics: EMNLP 2026
Sahar Abdelnabi | Aideen Fay | Ahmed Salem | Egor Zverev | Kai-Chieh Liao | Chi-Huang Liu | Chun Chih Kuo | Jannis Weigend | Danyael Manlangit | Alex Apostolov | Haris Umair | João Donato | Masayuki Kawakita | Athar Mahboob | Tran Huu Bach | Tsun-Han Chiang | Myeongjin Cho | Hajin Choi | Byeonghyeon Kim | Hyeonjin Lee | Benjamin Pannell | Conor McCauley | Mark Russinovich | Andrew Paverd | Giovanni Cherubin
Findings of the Association for Computational Linguistics: EMNLP 2026
Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100–1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack’s complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve <5% attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign–specifically trained against prompt injection–still yields 32% ASR, GPT-4o exhibits 58% ASR, and GPT-5 mini 47% ASR. We further demonstrate that attacks transfer across domains, achieving 37% ASR on proof-of-concept coding tasks and 28-45% ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations.