@inproceedings{abdelnabi-etal-2026-llmail,
title = "{LLM}ail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge",
author = "Abdelnabi, Sahar and
Fay, Aideen and
Salem, Ahmed and
Zverev, Egor and
Liao, Kai-Chieh and
Liu, Chi-Huang and
Kuo, Chun Chih and
Weigend, Jannis and
Manlangit, Danyael and
Apostolov, Alex and
Umair, Haris and
Donato, Jo{\~a}o and
Kawakita, Masayuki and
Mahboob, Athar and
Bach, Tran Huu and
Chiang, Tsun-Han and
Cho, Myeongjin and
Choi, Hajin and
Kim, Byeonghyeon and
Lee, Hyeonjin and
Pannell, Benjamin and
McCauley, Conor and
Russinovich, Mark and
Paverd, Andrew and
Cherubin, Giovanni",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.findings-emnlp.514/",
pages = "10757--10797",
ISBN = "979-8-89176-462-0",
abstract = "Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100{--}1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack{'}s complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve $<$5{\%} attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign{--}specifically trained against prompt injection{--}still yields 32{\%} ASR, GPT-4o exhibits 58{\%} ASR, and GPT-5 mini 47{\%} ASR. We further demonstrate that attacks transfer across domains, achieving 37{\%} ASR on proof-of-concept coding tasks and 28-45{\%} ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="abdelnabi-etal-2026-llmail">
<titleInfo>
<title>LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge</title>
</titleInfo>
<name type="personal">
<namePart type="given">Sahar</namePart>
<namePart type="family">Abdelnabi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Aideen</namePart>
<namePart type="family">Fay</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ahmed</namePart>
<namePart type="family">Salem</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Egor</namePart>
<namePart type="family">Zverev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Kai-Chieh</namePart>
<namePart type="family">Liao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chi-Huang</namePart>
<namePart type="family">Liu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chun</namePart>
<namePart type="given">Chih</namePart>
<namePart type="family">Kuo</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jannis</namePart>
<namePart type="family">Weigend</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Danyael</namePart>
<namePart type="family">Manlangit</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Alex</namePart>
<namePart type="family">Apostolov</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Haris</namePart>
<namePart type="family">Umair</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">João</namePart>
<namePart type="family">Donato</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Masayuki</namePart>
<namePart type="family">Kawakita</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Athar</namePart>
<namePart type="family">Mahboob</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tran</namePart>
<namePart type="given">Huu</namePart>
<namePart type="family">Bach</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tsun-Han</namePart>
<namePart type="family">Chiang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Myeongjin</namePart>
<namePart type="family">Cho</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Hajin</namePart>
<namePart type="family">Choi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Byeonghyeon</namePart>
<namePart type="family">Kim</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Hyeonjin</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Benjamin</namePart>
<namePart type="family">Pannell</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Conor</namePart>
<namePart type="family">McCauley</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Mark</namePart>
<namePart type="family">Russinovich</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Andrew</namePart>
<namePart type="family">Paverd</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Giovanni</namePart>
<namePart type="family">Cherubin</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-462-0</identifier>
</relatedItem>
<abstract>Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100–1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack’s complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve <5% attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign–specifically trained against prompt injection–still yields 32% ASR, GPT-4o exhibits 58% ASR, and GPT-5 mini 47% ASR. We further demonstrate that attacks transfer across domains, achieving 37% ASR on proof-of-concept coding tasks and 28-45% ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations.</abstract>
<identifier type="citekey">abdelnabi-etal-2026-llmail</identifier>
<location>
<url>https://aclanthology.org/2026.findings-emnlp.514/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>10757</start>
<end>10797</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge
%A Abdelnabi, Sahar
%A Fay, Aideen
%A Salem, Ahmed
%A Zverev, Egor
%A Liao, Kai-Chieh
%A Liu, Chi-Huang
%A Kuo, Chun Chih
%A Weigend, Jannis
%A Manlangit, Danyael
%A Apostolov, Alex
%A Umair, Haris
%A Donato, João
%A Kawakita, Masayuki
%A Mahboob, Athar
%A Bach, Tran Huu
%A Chiang, Tsun-Han
%A Cho, Myeongjin
%A Choi, Hajin
%A Kim, Byeonghyeon
%A Lee, Hyeonjin
%A Pannell, Benjamin
%A McCauley, Conor
%A Russinovich, Mark
%A Paverd, Andrew
%A Cherubin, Giovanni
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F abdelnabi-etal-2026-llmail
%X Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100–1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack’s complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve <5% attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign–specifically trained against prompt injection–still yields 32% ASR, GPT-4o exhibits 58% ASR, and GPT-5 mini 47% ASR. We further demonstrate that attacks transfer across domains, achieving 37% ASR on proof-of-concept coding tasks and 28-45% ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations.
%U https://aclanthology.org/2026.findings-emnlp.514/
%P 10757-10797
Markdown (Informal)
[LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge](https://aclanthology.org/2026.findings-emnlp.514/) (Abdelnabi et al., Findings 2026)
ACL
- Sahar Abdelnabi, Aideen Fay, Ahmed Salem, Egor Zverev, Kai-Chieh Liao, Chi-Huang Liu, Chun Chih Kuo, Jannis Weigend, Danyael Manlangit, Alex Apostolov, Haris Umair, João Donato, Masayuki Kawakita, Athar Mahboob, Tran Huu Bach, Tsun-Han Chiang, Myeongjin Cho, Hajin Choi, Byeonghyeon Kim, Hyeonjin Lee, Benjamin Pannell, Conor McCauley, Mark Russinovich, Andrew Paverd, and Giovanni Cherubin. 2026. LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 10757–10797, Budapest, Hungary. Association for Computational Linguistics.