Xueyu Hu
Author directory2026
GUI-PRA: Process Reward Agents for GUI Tasks
Tao Xiong | Xueyu Hu | Yurun Chen | Yuhang Liu | Changqiao Wu | Pengzhi Gao | Wei Liu | Jian Luan | Shengyu Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Tao Xiong | Xueyu Hu | Yurun Chen | Yuhang Liu | Changqiao Wu | Pengzhi Gao | Wei Liu | Jian Luan | Shengyu Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Long-horizon GUI automation remains challenging due to error accumulation over extended interaction sequences. Process Reward Models (PRMs) provide dense step-level supervision for mitigating error accumulation, yet standard PRMs are poorly suited to GUI verification. Standard PRM judgments often rely on superficial visual alignment rather than functional correctness, reflecting an evaluative knowledge gap caused by missing domain-specific adjudication logic. Standard PRMs also perform passive, single-pass visual assessment, which creates Visual Ambiguity when reliable judgment requires actively locating, parsing, or inspecting task-relevant UI evidence. We introduce GUI-PRA, a Process Reward Agent that transforms GUI process evaluation from passive scoring into active investigation. GUI-PRA couples Experience-Injected Criterion Synthesis, which distills generalized verification principles into state-specific criteria, with Criterion-Guided Autoregressive Perception, which uses these criteria to navigate multi-granularity visual tools and gather grounded evidence. On AndroidWorld and Mobile-MiniWoB++, GUI-PRA achieves improvements of 5.0% and 6.5% over standard PRMs on the Qwen-VL series, with Qwen3-VL attaining 54.74% success rate on AndroidWorld. On the offline OS-Critic Bench, GUI-PRA demonstrates strong competitiveness against fully trained critic models.
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
Yuhang Liu | Pengxiang Li | Zishu Wei | Congkai Xie | Xueyu Hu | Xinchen Xu | Shengyu Zhang | Xiaotian Han | Hongxia Yang | Fei Wu
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Yuhang Liu | Pengxiang Li | Zishu Wei | Congkai Xie | Xueyu Hu | Xinchen Xu | Shengyu Zhang | Xiaotian Han | Hongxia Yang | Fei Wu
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Graphical User Interface (GUI) Agents, powered by multimodal large language models (MLLMs), have shown great potential for task automation on computing devices such as computers and mobile phones. However, existing agents face challenges in multi-step reasoning and reliance on textual annotations, limiting their effectiveness. We introduce InfiGUIAgent, an MLLM-based GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1 enhances fundamental skills such as GUI understanding and grounding, while Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning skills using synthesized data to enable native reasoning abilities of the agents. InfiGUIAgent achieves competitive performance on several GUI benchmarks, highlighting the impact of native reasoning skills in enhancing GUI interaction for automation tasks.
DAC-Bench: A Decision-Aware Benchmark for Compositional Mobile GUI Tasks
Yuqing Zhang | Honghui Sheng | Xueyu Hu | Shengyu Zhang | Fei Wu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yuqing Zhang | Honghui Sheng | Xueyu Hu | Shengyu Zhang | Fei Wu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Mobile GUI agents powered by LMMs can perceive screens and follow instructions, yet existing benchmarks largely target short, linear workflows and step-level accuracy, offering limited insight into long-horizon planning and decision-making under branching structures. We present DAC-Bench, a decision-aware benchmark with compositional tasks comprising 830 episodes and 11,345 action steps across 35 applications on Android and iOS. Tasks are organized into Sequential, Conjunctive, Conditional, and Hierarchical structures, reflecting real-world multi-step and branching interaction patterns. To complement standard step-level evaluation, we introduce weighted longest common subsequence to capture length-sensitive progress and decision accuracy for branch correctness. Evaluations across 7 diverse agents show substantial performance degradation compared to prior benchmarks, with success rates dropping below 5% on 6–8 step tasks and branch accuracy averaging 38%, highlighting challenges in conditional decision-making. By exposing these failure modes, DAC-Bench provides a challenging and diagnostic benchmark for advancing decision-aware mobile GUI agents. Our code and dataset are available at: https://github.com/YuqingZhangMirror12/DAC-Bench.
2025
ClaimGen-CN: A Large-scale Chinese Dataset for Legal Claim Generation
Siying Zhou | Yiquan Wu | Hui Chen | Xueyu Hu | Kun Kuang | Adam Jatowt | Chunyan Zheng | Fei Wu
Findings of the Association for Computational Linguistics: EMNLP 2025
Siying Zhou | Yiquan Wu | Hui Chen | Xueyu Hu | Kun Kuang | Adam Jatowt | Chunyan Zheng | Fei Wu
Findings of the Association for Computational Linguistics: EMNLP 2025
Legal claims refer to the plaintiff’s demands in a case and are essential to guiding judicial reasoning and case resolution. While many works have focused on improving the efficiency of legal professionals, the research on helping non-professionals (e.g., plaintiffs) remains unexplored. This paper explores the problem of legal claim generation based on the given case’s facts. First, we construct ClaimGen-CN, the first dataset for Chinese legal claim generation task, from various real-world legal disputes. Additionally, we design an evaluation metric tailored for assessing the generated claims, which encompasses two essential dimensions: factuality and clarity. Building on this, we conduct a comprehensive zero-shot evaluation of state-of-the-art general and legal-domain large language models. Our findings highlight the limitations of the current models in factual precision and expressive clarity, pointing to the need for more targeted development in this domain. To encourage further exploration of this important task, we will make the dataset publicly available.
OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use
Xueyu Hu | Tao Xiong | Biao Yi | Zishu Wei | Ruixuan Xiao | Yurun Chen | Jiasheng Ye | Meiling Tao | Xiangxin Zhou | Ziyu Zhao | Yuhuai Li | Shengze Xu | Shenzhi Wang | Xinchen Xu | Shuofei Qiao | Zhaokai Wang | Kun Kuang | Tieyong Zeng | Liang Wang | Jiwei Li | Yuchen Eleanor Jiang | Wangchunshu Zhou | Guoyin Wang | Keting Yin | Zhou Zhao | Hongxia Yang | Fan Wu | Shengyu Zhang | Fei Wu
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Xueyu Hu | Tao Xiong | Biao Yi | Zishu Wei | Ruixuan Xiao | Yurun Chen | Jiasheng Ye | Meiling Tao | Xiangxin Zhou | Ziyu Zhao | Yuhuai Li | Shengze Xu | Shenzhi Wang | Xinchen Xu | Shuofei Qiao | Zhaokai Wang | Kun Kuang | Tieyong Zeng | Liang Wang | Jiwei Li | Yuchen Eleanor Jiang | Wangchunshu Zhou | Guoyin Wang | Keting Yin | Zhou Zhao | Hongxia Yang | Fan Wu | Shengyu Zhang | Fei Wu
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution of multi-modal large language models ((M)LLMs), this dream is closer to reality, as (M)LLM-based Agents using computers, mobile phones and web browsers by operating within the environments and interfaces (e.g., Graphical User Interface (GUI) and Command Line Interface (CLI)) provided by operating systems (OS) to automate tasks have significantly advanced. This paper presents a comprehensive survey on these advanced agents, designated as OS Agents. We begin by elucidating the fundamentals of OS Agents, exploring their key components and capabilities. We then examine methodologies for constructing OS Agents, focusing on domain-specific foundation models and agent frameworks. A detailed review of evaluation metrics and benchmarks highlights how OS Agents are assessed across diverse platforms and tasks. Finally, we discuss current challenges and identify promising directions for future research. An open-source GitHub repository is maintained as a dynamic resource to foster further innovation in this field.
Search
Fix author
Co-authors
- Shengyu Zhang 4
- Fei Wu 3
- Yurun Chen 2
- Kun Kuang 2
- Yuhang Liu 2
- Zishu Wei 2
- Tao Xiong 2
- Xinchen Xu 2
- Hongxia Yang 2
- Hui Chen 1
- Pengzhi Gao 1
- Xiaotian Han 1
- Adam Jatowt 1
- Yuchen Eleanor Jiang 1
- Jiwei Li 1
- Pengxiang Li 1
- Yuhuai Li 1
- Wei Liu 1
- Jian Luan 1
- Shuofei Qiao 1
- Honghui Sheng 1
- Meiling Tao 1
- Guoyin Wang 1
- Liang Wang 1
- Shenzhi Wang 1
- Zhaokai Wang 1
- Changqiao Wu 1
- Fan Wu 1
- Fei Wu 1
- Yiquan Wu 1
- Ruixuan Xiao 1
- Congkai Xie 1
- Shengze Xu 1
- Jiasheng Ye 1
- Biao Yi 1
- Keting Yin 1
- Tieyong Zeng 1
- Yuqing Zhang 1
- Zhou Zhao 1
- Ziyu Zhao 1
- Chunyan Zheng 1
- Siying Zhou 1
- Wangchunshu Zhou 1
- Xiangxin Zhou 1