Renhua Ding
Author directory2026
Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision–Language Agents
Renhua Ding | Xiao Yang | Zhengwei Fang | Jun Luo | Kun He
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Renhua Ding | Xiao Yang | Zhengwei Fang | Jun Luo | Kun He
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large vision-language models (LVLMs) are increasingly deployed as autonomous mobile agents that execute tasks through GUI interaction. However, their security under realistic deployment constraints remains insufficiently understood. Existing attacks typically rely on persistent visual manipulations, privileged system access, or iterative jailbreak interactions, limiting their practicality. In this work, we identify a previously overlooked discrepancy between human and agent interactions: automated mobile agents generate near-zero touch-contact signals that systematically differ from human touches. Based on this observation, we introduce agent-only perceptual injection, a new attack paradigm that exposes malicious prompts only during agent interactions while keeping them unexposed during normal user operation. To realize this attack, we propose a low-privilege framework that combines in-app prompt embedding, interaction-triggered activation, and HG-IDA*, an efficient one-shot jailbreak optimization method tailored to mobile UI constraints. Experiments on multiple Android applications and LVLM backends, such as GPT-4o, show that our attacks reliably hijack both agent planning and execution. Our method achieves up to 95.0% planning attack success rate and 82.5% execution attack success rate on commercial LVLM backends. These findings reveal a new interaction-level attack surface and highlight the need for provenance-aware defenses in mobile agent systems.