Pei Fu
Author directory2026
Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction
Yan Zhang | Pei Fu | Daiqing Wu | Huawen Shen | Ruoceng Zhang | Shaojie Zhang | Jiahui Yang | Yu Zhou | Can Ma | Zhenbo Luo | Jian Luan
Findings of the Association for Computational Linguistics: EMNLP 2026
Yan Zhang | Pei Fu | Daiqing Wu | Huawen Shen | Ruoceng Zhang | Shaojie Zhang | Jiahui Yang | Yu Zhou | Can Ma | Zhenbo Luo | Jian Luan
Findings of the Association for Computational Linguistics: EMNLP 2026
Graphical User Interface (GUI) Agents autonomously interact with software to fulfill user requests, where GUI navigation stands out as the most critical and challenging capability. Mastering this capability demands a complex synergy of step-wise decision-making, state-action alignment, and long-horizon planning. While directly mixing these corresponding navigation tasks seems intuitive to simultaneously acquire these skills, such a direct combination is severely bottlenecked by inconsistent optimization objectives and profound data heterogeneity. To overcome these barriers, we propose the MaP (stands for “Masked Trajectory Prediction”), a unified framework that seamlessly harmonizes divergent GUI navigation tasks. By modeling multi-turn GUI interactions as a trajectory and defining training objectives through component masking and prediction, MaP shifts the optimization from task-specific marginal distributions to a consistent objective. Furthermore, to handle the data heterogeneity across multiple navigation tasks, we design a role-aware adapter learning module that dynamically routes each token to a specialized representation space. Extensive experiments on five representative GUI navigation benchmarks demonstrate that MaP effectively mitigates gradient conflicts and significantly outperforms the direct mixture training, establishing a robust paradigm for multi-task GUI navigation.
Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models
Xiaoyi Yu | Enver Sangineto | Pei Fu | Fiorenzo Parascandolo | Wenhui Tan | Ruikang Zhang | Rita Cucchiara | Ruihua Song | Jian Luan
Findings of the Association for Computational Linguistics: EMNLP 2026
Xiaoyi Yu | Enver Sangineto | Pei Fu | Fiorenzo Parascandolo | Wenhui Tan | Ruikang Zhang | Rita Cucchiara | Ruihua Song | Jian Luan
Findings of the Association for Computational Linguistics: EMNLP 2026
Diffusion Large Language Models (dLLMs) have emerged as an efficient alternative to autoregressive models, yet aligning them via Reinforcement Learning (RL) requires likelihood surrogates estimated from masked reconstruction subproblems under a small Monte Carlo budget per rollout. Existing methods construct these subproblems by uniform random masking, leaving open the question of which subproblems to prioritize. We identify a systematic upstream/downstream structure in dLLM rollouts. Some tokens, when revealed, trigger large confidence changes in nearby undecoded positions; we call them upstream. Others induce only small local changes and are therefore downstream. We find masking downstream tokens yields substantially better-posed subproblems than masking upstream tokens, a phenomenon we term subproblem difficulty asymmetry. Based on the observation, we propose Informed Masking (IM), which derives a per-token priority score from the denoising trajectory at zero extra inference cost and biases mask sampling toward downstream tokens. IM is plug-and-play: when plugged into three state-of-the-art dLLM RL methods on LLaDA-8B-Instruct, it delivers up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks with improved training stability.
Doc-V*: Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
Yuanlei Zheng | Pei Fu | Hang Li | Ziyang Wang | Yuyi Zhang | Wenyu Ruan | Xiaojin Zhang | Zhongyu Wei | Zhenbo Luo | Jian Luan | Wei Chen | Xiang Bai
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yuanlei Zheng | Pei Fu | Hang Li | Ziyang Wang | Yuyi Zhang | Wenyu Ruan | Xiaojin Zhang | Zhongyu Wei | Zhenbo Luo | Jian Luan | Wei Chen | Xiang Bai
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents. Existing OCR-free methods face a trade-off between capacity and precision: end-to-end models scale poorly with document length, while visual retrieval-based pipelines are brittle and passive. We propose Doc-V*, an OCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation. Doc-V* begins with a thumbnail overview, then actively navigates via semantic retrieval and targeted page fetching, and aggregates evidence in a structured working memory for grounded reasoning. Trained by imitation learning from expert trajectories and further optimized with Group Relative Policy Optimization, Doc-V* balances answer accuracy with evidence-seeking efficiency. Across five benchmarks, Doc-V* outperforms open-source baselines and approaches proprietary models, improving out-of-domain performance by up to 47.9% over RAG baseline. Other results reveal effective evidence aggregation with selective attention, not increased input pages.
2025
Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
Pei Fu | Tongkun Guan | Zining Wang | Zhentao Guo | Chen Duan | Hao Sun | Boming Chen | Qianyi Jiang | Jiayao Ma | Kai Zhou | Junfeng Luo
Findings of the Association for Computational Linguistics: ACL 2025
Pei Fu | Tongkun Guan | Zining Wang | Zhentao Guo | Chen Duan | Hao Sun | Boming Chen | Qianyi Jiang | Jiayao Ma | Kai Zhou | Junfeng Luo
Findings of the Association for Computational Linguistics: ACL 2025
The recent emergence of Multi-modal Large Language Models (MLLMs) has introduced a new dimension to the Text-rich Image Understanding (TIU) field, with models demonstrating impressive and inspiring performance. However, their rapid evolution and widespread adoption have made it increasingly challenging to keep up with the latest advancements. To address this, we present a systematic and comprehensive survey to facilitate further research on TIU MLLMs. Initially, we outline the timeline, architecture, and pipeline of nearly all TIU MLLMs. Then, we review the performance of selected models on mainstream benchmarks. Finally, we explore promising directions, challenges, and limitations within the field.
Search
Fix author
Co-authors
- Jian Luan 3
- Zhenbo Luo 2
- Xiang Bai 1
- Boming Chen 1
- Wei Chen 1
- Rita Cucchiara 1
- Chen Duan 1
- Tongkun Guan 1
- Zhentao Guo 1
- Qianyi Jiang 1
- Hang Li 1
- Junfeng Luo 1
- Can Ma 1
- Jiayao Ma 1
- Fiorenzo Parascandolo 1
- Wenyu Ruan 1
- Enver Sangineto 1
- Huawen Shen 1
- Ruihua Song 1
- Hao Sun 1
- Wenhui Tan 1
- Zining Wang 1
- Ziyang Wang 1
- Zhongyu Wei (魏忠钰) 1
- Daiqing Wu 1
- Jiahui Yang 1
- Xiaoyi Yu 1
- Ruikang Zhang 1
- Ruoceng Zhang 1
- Shaojie Zhang 1
- Xiaojin Zhang 1
- Yan Zhang 1
- Yuyi Zhang 1
- Yuanlei Zheng 1
- Kai Zhou 1
- Yu Zhou 1