Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot

Yuyang Dai, Yushun Dong


Abstract
Large language models deployed as commercial APIs are vulnerable to model extraction attacks, while existing defenses either act too late or degrade utility for legitimate users. We propose Knowledge Trap, a defense that redirects extraction attacks toward low-transferability knowledge through a Honeypot Knowledge Graph (HKG) and breadcrumb-guided exploration. Instead of blocking queries or perturbing outputs, Knowledge Trap consumes the attacker’s limited query budget on knowledge with negligible downstream utility while preserving benign-user performance. Experiments in medical and financial domains show that Knowledge Trap reduces surrogate Agreement by 6.2% on average without degrading legitimate-user accuracy, outperforming existing defenses that impose measurable user impact. These results suggest that defending knowledge-space traversal is a practical direction for mitigating LLM extraction attacks.
Anthology ID:
2026.emnlp-main.411
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
9428–9449
Language:
URL:
https://aclanthology.org/2026.emnlp-main.411/
DOI:
Bibkey:
Cite (ACL):
Yuyang Dai and Yushun Dong. 2026. Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 9428–9449, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot (Dai & Dong, EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.411.pdf