Zhihao Qin
Author directory2026
Plan the Semantics, Then Render the Signal: A Hybrid State-Space Model for Incomplete Multimodal Learning
Ruiting Dai | Zhihao Qin | Yilin Chen | Xianhua Yan | Lisi Mo | Tao He
Findings of the Association for Computational Linguistics: EMNLP 2026
Ruiting Dai | Zhihao Qin | Yilin Chen | Xianhua Yan | Lisi Mo | Tao He
Findings of the Association for Computational Linguistics: EMNLP 2026
Incomplete multimodal learning seeks robust perception when modalities are absent at training or test time, using available streams to recover the missing ones. Most methods cast this as direct regression in a fixed continuous latent, which (i) collapses many valid solutions into an averaged reconstruction when observations underdetermine the target, and (ii) suffers semantic drift when cross-modal relationships shift with context. We argue that completion should first resolve what is missing and only then render how it appears. We realize this as Semantic-first Completion, instantiated by a Hybrid State-Space Model (HySSM) that decouples semantic intent from signal realization. A frozen omni-modal LLM plans over observation-only evidence, generating an evidence-grounded Markovian trajectory of discrete semantic states that act as a compact information bottleneck for stable cross-modal reasoning. The plan is projected into a coordinate-wise control field that drift-guides a conditional diffusion model, synthesizing missing signals or feature streams as diverse and semantically consistent completions. Across MNIST, Hateful Memes, and CMU-MOSEI, HySSM surpasses recent generative baselines and, under agnostic masking shift, remains robust up to 90% missingness.