Zijie Zhang

Author directory

Other people with similar names: Zijie Zhang

Unverified author pages with similar names: Zijie Zhang


2026

Long-video Omni assistants often fail not by inventing content, but by misbinding real evidence: they hear the right utterance and see the right event, yet attach it to the wrong speaker, moment, or modality. These almost-true errors evade standard video QA because local evidence remains valid, so item-level scoring can reward both a supported claim and its near-counterfactual. We introduce a counterfactual event-binding protocol that constructs paired supported/counterfactual claims from the same audio-visual event evidence and evaluates them by strict-pair accuracy. We instantiate it as OmniHalluc-L, to our knowledge the first benchmark for long-video Omni hallucination, with 3600 single-claim QA items from 638 long-form videos averaging 24.16 minutes and covering 256.87 hours. Under this protocol, open-weight Omni models remain weak at pair-level binding: Qwen2.5-Omni-7B reaches 32.06% and Qwen3-Omni-Instruct reaches 41.55%, versus 76.54% for a closed-source reference. To narrow this gap, we propose MPRC (Modality-Perturbation Reliability Calibration), a frozen-backbone calibrator that combines native confidence with response shifts from selected audio-negative probes. MPRC raises these two models to 36.22% and 51.09%, respectively, and improves target-adapted MCQ accuracy on OmniVideoBench and WorldSense. The code is available at https://github.com/ZexDong/omnihallucL-mprc.
Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose **DR3-Eval**, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR3-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR3-Agent based on multiple state-of-the-art language models demonstrate that DR3-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control.
Omni-modal video captioning is not merely combining visual captioning with audio transcription: a useful caption must describe how visual actions, speech, music, and sound effects co-evolve over time. Existing large multimodal models often fail at this relational step, treating audio and visual streams as loosely coupled observations, relying heavily on automatic speech recognition, and under-specifying non-speech sounds and their links to visual events. We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal training corpus generated by a decoupled-then-fused pipeline that anchors visual and acoustic evidence independently before composing temporally grounded omni-modal captions. Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline captioning ability, while a sample-efficient reinforcement learning stage uses hybrid rewards to optimize acoustic completeness and audio-visual synergy. Our scaling analysis shows that reinforcement learning brings larger gains than simply increasing SFT data. Third, we introduce AVSCapBench, a human-curated benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall. Experiments on AVSCapBench and external benchmarks show that AVSCap-7B improves non-speech audio coverage and cross-modal binding, delivering the best overall performance among evaluated open-source models.