Haibo Qiu
Author directory2026
MobileDreamer: Generative Sketch World Model for GUI Agent
Yilin Cao | Yufeng Zhong | Zhixiong Zeng | Liming Zheng | Jing Huang | Haibo Qiu | Peng Shi | Wenji Mao
Findings of the Association for Computational Linguistics: EMNLP 2026
Yilin Cao | Yufeng Zhong | Zhixiong Zeng | Liming Zheng | Jing Huang | Haibo Qiu | Peng Shi | Wenji Mao
Findings of the Association for Computational Linguistics: EMNLP 2026
Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from the current screen, which limits their performance on long-horizon tasks. To address this limitation, a natural approach is to use the forecasting ability of world models to support better decision-making. This is challenging because existing world models often lack spatial awareness when predicting post-action states and are too expensive for practical deployment. In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. It consists of textual sketch world model and rollout imagination for GUI agent. Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. The rollout imagination strategy for the GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. Empirical results on Android World demonstrate that MobileDreamer consistently improves the task success rate across multiple base models, achieving an average performance gain of 5.3%. World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements.
VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning
Xuanle Zhao | Deyang Jiang | Zhixiong Zeng | Lei Chen | Haoyue Yang | Haibo Qiu | Jing Huang | Yufeng Zhong | Liming Zheng | Yilin Cao | Lin Ma
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Xuanle Zhao | Deyang Jiang | Zhixiong Zeng | Lei Chen | Haoyue Yang | Haibo Qiu | Jing Huang | Yufeng Zhong | Liming Zheng | Yilin Cao | Lin Ma
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
While recent specialized multimodal code generation models excel in tasks like chart-to-code generation, their reliance on single-task training limits generalization and hinders the development of VIsioN Code Intelligence. In this work, we introduce VinciCoder, a unified framework designed for generalized multimodal code generation. We first curate a large-scale SFT corpus comprising 1.3M direct generation pairs and 300k visual-based refinement tasks. This composition fosters self-refinement capabilities, enabling the model to directly rectify code to align with input images. Subsequently, we propose coarse-to-fine Visual Reinforcement Learning (ViRL) to overcome the brittleness of textual metrics in handling semantically equivalent but syntactically diverse code. By quantifying visual similarity across multi-scale patches, ViRL provides an implementation-agnostic reward mechanism that ensures high-fidelity alignment between rendered outputs and input visuals. Extensive experimental results across diverse benchmarks demonstrate that VinciCoder achieves superior performance, while comprehensive ablation studies validate the effectiveness of our proposed ViRL strategy. The data, code and model are available at https://github.com/DocTron-hub/VinciCoder.
Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation
Jiawei Zhou | Chi Zhang | Xiang Feng | Qiming Zhang | Haibo Qiu | Lihuo He | Dengpan Ye | Xinbo Gao | Jing Zhang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Jiawei Zhou | Chi Zhang | Xiang Feng | Qiming Zhang | Haibo Qiu | Lihuo He | Dengpan Ye | Xinbo Gao | Jing Zhang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
We present Omni-I2C, a comprehensive benchmark designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code. We argue that this task represents a non-trivial challenge for the current generation of LMMs: it demands an unprecedented synergy between high-fidelity visual perception—to parse intricate spatial hierarchies and symbolic details—and precise generative expression—to synthesize syntactically sound and logically consistent code. Unlike traditional descriptive tasks, Omni-I2C requires a holistic understanding where any minor perceptual hallucination or coding error leads to a complete failure in visual reconstruction. Omni-I2C features 1130 meticulously curated samples, defined by its breadth across subjects, image modalities, and programming languages. By incorporating authentic user-sourced cases, the benchmark spans a vast spectrum of digital content—from scientific visualizations to complex symbolic notations—each paired with executable reference code. To complement this diversity, our evaluation framework provides necessary depth; by decoupling performance into perceptual fidelity and symbolic precision, it transcends surface-level accuracy to expose the granular structural failures and reasoning bottlenecks of current LMMs. Our evaluation reveals a substantial performance gap among leading LMMs; even state-of-the-art models struggle to preserve structural integrity in complex scenarios, underscoring that multimodal code generation remains a formidable challenge. Data and code are available at https://github.com/MiliLab/Omni-I2C.