Shortcut-Controlled Evaluation for AI-Generated Text Detection

Hee-Young Jung, Jae-Eon Ahn, Seong-Jin Park, Kang-Min Kim


Abstract
Large language models (LLMs) are increasingly employed in writing assistance for online text, making AI-generated text (AIGT) detection a critical challenge for managing text provenance. However, it remains unclear whether the high detection performance reported in existing AIGT benchmarks reflects a detector’s ability to capture essential differences between human-written and AI-generated text. Rather, this high performance might be driven by easily measurable shortcuts in benchmark settings. To address this question, we decomposed the detection signals in AIGT and systematically analyzed the role of shortcuts in existing benchmark settings. Our analysis reveals that a simple logistic regression classifier trained solely on surface shortcut features (e.g., formatting style and lexical usage) achieves strong classification performance without relying on any semantic signals. Furthermore, the consistent performance degradation of existing detectors when such shortcuts are removed suggests that current AIGT evaluations are influenced by shortcut signals. Motivated by these findings, we present AIGT-SignalBench, a shortcut-controlled benchmark designed to reflect realistic AI-assisted writing scenarios.
Anthology ID:
2026.emnlp-main.2177
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
49501–49530
Language:
URL:
https://aclanthology.org/2026.emnlp-main.2177/
DOI:
Bibkey:
Cite (ACL):
Hee-Young Jung, Jae-Eon Ahn, Seong-Jin Park, and Kang-Min Kim. 2026. Shortcut-Controlled Evaluation for AI-Generated Text Detection. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 49501–49530, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Shortcut-Controlled Evaluation for AI-Generated Text Detection (Jung et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.2177.pdf