Xianfei Li

Author directory

2026

Object hallucination in Multimodal Large Language Models (MLLMs) often appears in two forms: omitting objects that are present in the image and generating objects that are absent. Existing training-free mitigation methods usually apply a uniform intervention to both cases. We find that this strategy leads to an imbalanced effect: it can reduce some omission errors, but often introduces new fabrications. To understand this behavior, we analyze spatial attention patterns and layer-wise attention dynamics during generation. Our results suggest that omissions often occur when the model attends to the relevant visual regions but gradually loses visual confidence during decoding, whereas fabrications are more closely related to deep-layer semantic priors overriding visual evidence. Based on this observation, we propose Visual Potential Field Calibration (VPFC), a training-free hidden-state calibration method. VPFC locates an intervention region using the spatial centroid of high-confidence visual tokens and applies a lightweight hidden-state update to adjust the model’s generative confidence. Experiments on multiple hallucination benchmarks show that VPFC consistently improves over existing training-free methods while maintaining inference efficiency close to standard greedy decoding.

2025

User simulators are crucial for replicating human interactions with dialogue systems, supporting both collaborative training and automatic evaluation, especially for large language models (LLMs). However, current role-playing methods face challenges such as a lack of utterance-level authenticity and user-level diversity, often hindered by role confusion and dependence on predefined profiles of well-known figures. In contrast, direct simulation focuses solely on text, neglecting implicit user traits like personality and conversation-level consistency. To address these issues, we introduce the User Simulator with Implicit Profiles (USP), a framework that infers implicit user profiles from human-machine interactions to simulate personalized and realistic dialogues. We first develop an LLM-driven extractor with a comprehensive profile schema, then refine the simulation using conditional supervised fine-tuning and reinforcement learning with cycle consistency, optimizing at both the utterance and conversation levels. Finally, a diverse profile sampler captures the distribution of real-world user profiles. Experimental results show that USP outperforms strong baselines in terms of authenticity and diversity while maintaining comparable consistency. Additionally, using USP to evaluate LLM on dynamic multi-turn aligns well with mainstream benchmarks, demonstrating its effectiveness in real-world applications.