Zhengdong Hu

Author directory

2026

This paper explores agentic 3D spatial understanding, i.e., MLLM agents performing indoor 3D reasoning through tool use. Existing methods often misuse tools and exhibit biased tool preferences, leaving the agentic paradigm with only marginal gains over non-agentic strategies. We reveal that 3D spatial reasoning tasks are heterogeneous across scenes, while these agents apply a uniform tool-use strategy to all scenes rather than selecting tools according to the specific scene and task. To address this, we propose Skill-3D, a framework that learns self-evolving scene-aware skills. Specifically, Skill-3D identifies the task scene and records the agent’s tool-use trajectory into a Scene Memory, where successful trajectories from similar scenes are aggregated and distilled into a reusable scene-aware skill, with failed ones attached to the skill as lessons. During training, once a similar scene recurs, the corresponding skill is injected to guide the agent, producing new trajectories whose successes and failures further refine the skill, forming a loop in which the memory and the skill library co-evolve. Experiments show that Skill-3D substantially improves tool utilization in 3D spatial reasoning (from 39%to 78% on VSI-Bench), driving the agent toward correct and sufficient tool use. For instance, it improves Gemini-3-Flash by 67% on MMSI-Bench. Furthermore, we conduct agentic reinforcement learning over skill-guided trajectories, which boosts Qwen3-VL-8B by 60% on VSI-Bench.
Reinforcement Learning (RL) has proven effective in eliciting reasoning abilities of Multimodal Large Language Models (MLLMs) and is emerging in document understanding. However, existing approaches usually lack fine-grained region-level grounding supervision, while the key challenge in document reasoning lies in linking visual evidence within precise regions directly to question–answer semantics. We introduce RegionDoc-R1, a novel framework for document understanding that enhances MLLMs’ reasoning with step-wise feedback. Instead of previous OCR-based text positions, we encourage the model to predict question-relevant Regions of Interest (RoIs) and ground the answer in these regions. Specifically, we first propose the Region-Aware Group Relative Policy Optimization (RA-GRPO), which encourages models to utilize region-level spatial information in documents for reasoning. Correspondingly, we construct a hybrid training corpus, named SR-Doc, containing Semantic Reasoning (SR) examples enriched with cross-page and region-level reasoning layout annotations. Meanwhile, we also introduce an Adaptive Chain-of-Thought (Ada-CoT) strategy, which dynamically adjusts the reasoning process according to different tasks, enabling more efficient and flexible step-wise document understanding. Experiments on several document reasoning benchmarks demonstrate that RegionDoc-R1 achieves state-of-the-art performance across tasks such as form document understanding, table-based QA, and layout-sensitive information extraction.