Bo Gao
Author directoryOther people with similar names: Bo Gao, Bo Gao
2026
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Yifan Xiang | Zhenxi Zhang | Bin Li | Yixuan Weng | Bo Gao | Shoujun Zhou | Yangfan He | Yilin Yuan | Keqin Li
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yifan Xiang | Zhenxi Zhang | Bin Li | Yixuan Weng | Bo Gao | Shoujun Zhou | Yangfan He | Yilin Yuan | Keqin Li
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance across a wide range of multimodal tasks. Recent advances in personalized MLLMs enable effective capture of user-specific concepts, supporting both recognition of personalized concepts and contextual captioning. However, humans typically explore and reason over relations among objects and individuals, transcending surface-level information to achieve more personalized and contextual understanding. To this end, existing methods may face three main limitations: (1) Their training data lacks multi-object sets in which relations among objects are learnable, (2) Existing models often neglect the connections between different personalized concepts, thereby failing to perform reasoning over them, (3) Their experiments mainly focus on a single personalized concept, where evaluations are limited to recognition and captioning tasks. To address the limitations, (i) We present a new dataset named ReGraP, consisting of 120 sets of personalized knowledge. Each set includes images, Knowledge Graphs (KGs), and Chain-of-Thought Question-Answering (CoT QA) pairs derived from the KGs, enabling more structured and sophisticated reasoning pathways. (ii) We propose Reasoning enabled Graph-based Personalized Large Language and Vision Assistant (ReGraP-LLaVA), an MLLM trained with the corresponding KGs and CoT QA pairs, where soft and/or hard graph prompting methods are designed to align KGs within the model’s semantic space. (iii) We establish the ReGraP Benchmark, which contains diverse task types: Multiple-Choice, Fill-in-the-blank, True/False, and Descriptive questions in both open- and closed-ended settings. The proposed benchmark is designed to evaluate the relational reasoning and knowledge-connection capability of personalized MLLMs. We conduct experiments on the proposed ReGraP-LLaVA and other competitive MLLMs. Results show that the proposed model not only learns personalized knowledge but also performs relational reasoning in responses, achieving the SoTA performance compared with the competitive methods. All the codes and datasets are released at: https://anonymous.4open.science/r/ReGraP.
Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing
Miao Wang | Yuling Shi | Yijiang Li | Yeheng Chen | Xiaodong Gu | Bin Li | Bo Gao | Wang Jun | Zengxin Han | Jingtong wu | Yaduan Ruan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Miao Wang | Yuling Shi | Yijiang Li | Yeheng Chen | Xiaodong Gu | Bin Li | Bo Gao | Wang Jun | Zengxin Han | Jingtong wu | Yaduan Ruan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Text-based role-playing models can imitate character styles, but often fail to capture scene atmosphere and evolving tension, which are crucial for immersive applications such as VR games and interactive narratives. We study video-grounded role-playing dialogue and introduce EBM-RL (Eye-Brain-Mouth Reinforcement Learning), a decoupled GRPO-based framework that separates observation ( lt;perception gt;), reasoning ( lt;think gt;), and utterance generation ( lt;answer gt;). This design mimics the human See-Think-Speak process, enabling the model to ground dialogue in visual perception before reasoning and response generation. To optimize this See-Think-Speak process, EBM-RL integrates complementary rewards for scene-text alignment, perceptual-cognitive utility, answer faithfulness, and format consistency. Extensive experiments show that EBM-RL substantially outperforms text-only role-playing baselines and comparable-scale VLM baselines, while remaining competitive with stronger larger-scale VLMs on visual grounding and situational persona metrics. Moreover, EBM-RL demonstrates strong zero-shot transfer to out-of-domain VideoQA benchmarks without additional fine-tuning. We also release an open-source dataset for video-grounded role-playing dialogue.