Jun Lan
Author directory2026
Causal Probing for Internal Visual Representations in Multimodal Large Language Models
Zehao Deng | Tianjie Ju | Zheng Wu | Liangbo He | Jun Lan | Huijia Zhu | Weiqiang Wang | Zhuosheng Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zehao Deng | Tianjie Ju | Zheng Wu | Liangbo He | Jun Lan | Huijia Zhu | Weiqiang Wang | Zhuosheng Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Despite the remarkable success of Multimodal Large Language Models (MLLMs) across diverse tasks, the internal mechanisms governing how they encode and ground distinct visual concepts remain poorly understood. To unravel these mechanisms, we propose a causal framework based on activation steering to actively probe and manipulate internal visual representations. Through systematic intervention across four visual concept categories, our results reveal a divergence in concept encoding: entity knowledge is distinctively localized, whereas abstract concepts are globally distributed across the network. Critically, this divergence uncovers a mechanistic driver of scaling laws: increasing model depth is indispensable for encoding distributed and complex abstract concepts, whereas entities maintain a consistently high degree of localization. Furthermore, reverse steering uncovers that blocking explicit output triggers a surge in latent activations, exposing a compensatory mechanism between perception and generation. Finally, by extending our analysis to visual reasoning, we expose a disconnect between perception and reasoning: although MLLMs successfully recognize geometric relations, they treat them merely as static visual features, failing to trigger the procedural execution necessary for problem-solving.
RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection
Yan Hong | Wei Li | Kedong Xiu | Jun Lan | Shuheng Zhou | Zhongcai Lyu | Huijia Zhu | Weiqiang Wang | Jianfu Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yan Hong | Wei Li | Kedong Xiu | Jun Lan | Shuheng Zhou | Zhongcai Lyu | Huijia Zhu | Weiqiang Wang | Jianfu Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Knowledge injection updates pretrained MLLMs with new factual or domain-specific knowledge, but fitting full authoritative answers can cause drift in non-updated behavior. Online distillation limits this drift by training on model-generated rollouts, yet uniform reference-conditioned distillation gives coarse supervision: it can under-emphasize reference-supported rollout tokens and supervise omitted facts only indirectly. We introduce RoCo-ACE, a rollout-conditioned online distillation objective for knowledge injection: RoCo uses same-rollout reference-free/reference-conditioned likelihood contrast to reallocate additional distillation weight to reference-supported rollout tokens, while ACE adds sparse reference-side anchored correction for authoritative anchors omitted from the rollout, without full-answer imitation. Across three knowledge-injection settings, six retention benchmarks, multiple baselines and base models, RoCo-ACE achieves the best injected-knowledge accuracy among compared methods while keeping evaluated retention close to the base model.
Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents
Zhaoyuan Huang | Tianjie Ju | Pengzhou Cheng | Zheng Wu | Yansi Li | Chuanbiao Song | Jun Lan | Huijia Zhu | Weiqiang Wang | Zhuosheng Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zhaoyuan Huang | Tianjie Ju | Pengzhou Cheng | Zheng Wu | Yansi Li | Chuanbiao Song | Jun Lan | Huijia Zhu | Weiqiang Wang | Zhuosheng Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Graphical user interface (GUI) agents are increasingly used to execute natural-language instructions on user interfaces, yet real users may issue infeasible instructions due to benign mistakes. A reliable agent should not only know how to act, but also when not to act. In this work, we introduce ConflictGUI, a benchmark covering instruction-internal conflicts and instruction-GUI context conflicts to study conflict-aware termination. Our evaluation reveals severe execution-biased over-compliance: agents that perform well on feasible tasks often continue to execute blindly under conflicting instructions. To mitigate this behavior, we propose ConflictGuard, an inference-time framework that aligns an agent’s feasibility awareness with its action generation. ConflictGuard contains two coupled components: a feasibility verification protocol that guides the agent to assess instruction logic and GUI-side evidence before acting, and a conditional action modulation mechanism that steers agent’s over-compliance into termination-oriented behavior. Experiments across five widely-used agents demonstrate that ConflictGuard improves average conflict task success rate significantly, while preserving normal GUI-task performance. These results validate that a lightweight inference-time intervention can substantially boost GUI Agent’s competence to identify inappropriate execution scenarios and refrain from unnecessary actions.