Xusheng Xiao
Author directory2026
Your Harness is Not Secure: Benchmarking Real-world Threat of Command Line Interface Agent
Weidi Luo | Qiming Zhang | Tianyu Lu | Xiaogeng Liu | Bin Hu | Chiu Hung Chun | Siyuan Ma | Yizhe Zhang | Xusheng Xiao | Yinzhi Cao | Zhen Xiang | Chaowei Xiao
Findings of the Association for Computational Linguistics: EMNLP 2026
Weidi Luo | Qiming Zhang | Tianyu Lu | Xiaogeng Liu | Bin Hu | Chiu Hung Chun | Siyuan Ma | Yizhe Zhang | Xusheng Xiao | Yinzhi Cao | Zhen Xiang | Chaowei Xiao
Findings of the Association for Computational Linguistics: EMNLP 2026
Command-line interface (CLI) agents powered by large language models (LLMs) can interpret natural-language requests, plan multi-step tasks, execute shell commands, and modify files and system state. As these agents are increasingly used for operating-system (OS) workflows, it is important to evaluate whether they can be misused to carry out security-relevant operations. Existing benchmarks often lack an attacker-knowledge model grounded in tactics, techniques, and procedures (TTPs), provide limited coverage of end-to-end kill chains, rely on simplified single-host environments, or use LLM-as-a-judge for success evaluation. We introduce ADVCLI, an MITRE ATT CK-aligned benchmark for evaluating OS-level misuse risks of CLI agents in a controlled multi-host sandbox. ADVCLI contains 140 tasks: 40 direct malicious requests, 74 TTP-based tasks, and 26 end-to-end kill chains. Each task is paired with deterministic hard-coded verification protocols that check whether the requested OS-level effect is realized. We evaluate seven CLI Agents and products built on nine foundation models, including ReAct, OpenClaw, OpenAI Agent SDK, Claude Code, Gemini CLI, Cursor CLI, and Cursor IDE. Results show that current CLI frequently proceed beyond refusal and can complete a non-negligible fraction of malicious OS-level tasks, especially when requests include TTP-style attacker knowledge. ADVCLI provides a reproducible testbed for evaluating these risks and for developing stronger safety mechanisms for tool-using CLI agents in the future.
Agent+P: Guiding UI Agents via Symbolic Planning
Shang Ma | Xusheng Xiao | Yanfang Ye
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Shang Ma | Xusheng Xiao | Yanfang Ye
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Large Language Model (LLM)-based UI agents show great promise for UI automation but of- ten hallucinate in long-horizon tasks due to their lack of understanding of the global UI transition structure. To address this, we in- troduce AGENT+P, a novel framework that leverages symbolic planning to guide LLM- based UI agents. Specifically, we model an app’s UI transition structure as a UI Transi- tion Graph (UTG), which allows us to refor- mulate the UI automation task as a pathfinding problem on the UTG. This further enables an off-the-shelf symbolic planner to generate a provably correct and optimal high-level plan, preventing the agent from redundant explo- ration and guiding the agent to achieve the au- tomation goals. AGENT+P is designed as a plug-and-play framework to enhance existing UI agents. Evaluation on the AndroidWorld benchmark demonstrates that AGENT+P im- proves the success rates of state-of-the-art UI agents by up to 14.32% and reduces the ac- tion steps by 37.70%. Our code is available at: https://github.com/KiteFlyKid/agentp.
2025
PsyScam: A Benchmark for Psychological Techniques in Real-World Scams
Shang Ma | Tianyi Ma | Jiahao Liu | Wei Song | Zhenkai Liang | Xusheng Xiao | Yanfang Ye
Findings of the Association for Computational Linguistics: EMNLP 2025
Shang Ma | Tianyi Ma | Jiahao Liu | Wei Song | Zhenkai Liang | Xusheng Xiao | Yanfang Ye
Findings of the Association for Computational Linguistics: EMNLP 2025
Over the years, online scams have grown dramatically,with nearly 50% of global consumersencountering scam attempts each week.These scams cause not only significant financiallosses to individuals and businesses, butalso lasting psychological trauma, largely dueto scammers’ strategic employment of psychologicaltechniques (PTs) to manipulate victims.Meanwhile, scammers continually evolve theirtactics by leveraging advances in Large LanguageModels (LLMs) to generate diverse scamvariants that easily bypass existing defenses.To address this pressing problem, we introducePsyScam, a benchmark designed to systematicallycapture the PTs employed in real-worldscam reports, and investigate how LLMs canbe utilized to generate variants of scams basedon the PTs and the contexts provided by thesescams. Specifically, we collect a wide range ofscam reports and ground its annotations of employedPTs in well-established cognitive andpsychological theories. We further demonstrateLLMs’ capabilities in generating through twodownstream tasks: scam completion, and scamaugmentation. Experimental results show thatPsyScam presents significant challenges toexisting models in both detecting and generatingscam content based on the PTs used byreal-world scammers. Our code and dataset areavailable.
Search