Renhua Ding

Author directory

2026

Large vision-language models (LVLMs) are increasingly deployed as autonomous mobile agents that execute tasks through GUI interaction. However, their security under realistic deployment constraints remains insufficiently understood. Existing attacks typically rely on persistent visual manipulations, privileged system access, or iterative jailbreak interactions, limiting their practicality. In this work, we identify a previously overlooked discrepancy between human and agent interactions: automated mobile agents generate near-zero touch-contact signals that systematically differ from human touches. Based on this observation, we introduce agent-only perceptual injection, a new attack paradigm that exposes malicious prompts only during agent interactions while keeping them unexposed during normal user operation. To realize this attack, we propose a low-privilege framework that combines in-app prompt embedding, interaction-triggered activation, and HG-IDA*, an efficient one-shot jailbreak optimization method tailored to mobile UI constraints. Experiments on multiple Android applications and LVLM backends, such as GPT-4o, show that our attacks reliably hijack both agent planning and execution. Our method achieves up to 95.0% planning attack success rate and 82.5% execution attack success rate on commercial LVLM backends. These findings reveal a new interaction-level attack surface and highlight the need for provenance-aware defenses in mobile agent systems.