Canjun Yang

Author directory

2026

Video understanding is rapidly evolving toward omni-modal fusion, yet Multimodal Large Language Models (MLLMs) still struggle with complex audio-visual reasoning and semantic alignment. Current paradigms often rely on independent modal projectors to map features into the LLM space, an architecture that frequently overlooks intrinsic modal heterogeneity and inadvertently reinforces vision-centric bias by avoiding early, direct semantic interaction between native cross-modal representations. To address these challenges, we propose SynAV, a novel MLLM architecture designed for language-grounded multimodal reasoning. Our framework introduces a two-stage hierarchical cross-modal semantic interaction mechanism: it first performs alignment within the textual semantic space, and subsequently executes a visual-to-audio refinement that bridges semantic gaps and captures temporal dynamics. Furthermore, we employ a frame-by-frame interleaved audio-visual token arrangement enriched with explicit textual timestamps. This strategy acts as a temporal-semantic anchoring mechanism, allowing the LLM to better perceive the synchrony between modalities and seamlessly conduct long-sequence logical reasoning. Our model jointly optimizes temporal grounding and dense captioning tasks in an autoregressive manner. Extensive experiments across 11 benchmarks demonstrate that SynAV significantly outperforms state-of-the-art methods, achieving a superior performance in complex audio-visual language reasoning.