Jiaming Han

Author directory

2026

Recent progress in video generative models has enabled the creation of high-quality videos from multimodal prompts that combine text and images. While these systems offer greater controllability, they also introduce new safety risks, as harmful content can arise from individual modalities or their interaction. Existing safety methods are often limited to text-only inputs, require the user to manually specify the risk category in advance, or only audit content after generation, making them unable to proactively address compositional multimodal risks. We present ConceptGuard, a safeguard framework that proactively detects and mitigates unsafe semantics in multimodal video generation before harmful content is produced. ConceptGuard operates in two stages. First, a contrastive detection module fuses image and text inputs and computes their similarity against the risk concept set , automatically retrieving the top-k concepts most relevant to the input without requiring any prior assumption about the risk category. Second, a semantic suppression mechanism uses these identified concepts to remove unsafe semantics from both the text and the visual input during early generation steps. To support training and evaluation, we introduce ConceptRisk, a large-scale dataset for multimodal risk detection, and T2VSafetyBench-TI2V, the first benchmark for evaluating safety in Text-and-Image-to-Video generation. Experiments on both benchmarks show that ConceptGuard outperforms existing baselines in both risk detection and safe video generation.
Automating the transformation of user interface (UI) designs into front-end code holds significant promise for accelerating software development and democratizing design workflows. While multimodal large language models (MLLMs) can translate images to code, they often fail on complex UIs, struggling to unify visual perception, layout planning, and code synthesis within a single monolithic model, which leads to frequent perception and planning errors. To address this, we propose ScreenCoder, a modular multi-agent framework that decomposes the task into three interpretable stages: grounding, planning, and generation. By assigning these distinct responsibilities to specialized agents, our framework achieves significantly higher robustness and fidelity than end-to-end approaches. Furthermore, ScreenCoder serves as a scalable data engine, enabling us to generate high-quality image-code pairs. We use this data to fine-tune open-source MLLM via a dual-stage pipeline of supervised fine-tuning and reinforcement learning, demonstrating substantial gains in its UI generation capabilities. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in layout accuracy, structural coherence, and code correctness.
Recent multimodal large language models (MLLMs), such as GPT-4o, Gemini 1.5/2.5 Pro, and Reka Core, have advanced audio-visual reasoning capabilities, achieving strong performance in tasks like cross-modal understanding and generation. However, our DeafTest uncovers unanticipated failures: most of the state-of-the-art MLLMs struggle with very simple audio tasks, such as distinguishing louder sounds or sound counting. This raises a fundamental question—does a deficiency in low-level audio perception constrain higher-level audio-visual reasoning? To address this, we introduce AV-Odyssey Bench—a comprehensive benchmark of 4,555 meticulously designed problems that integrate text, audio, and visual modalities. Each task requires models to unify cross-modal reasoning, leveraging synchronized audio-visual cues to infer solutions. By structuring questions as multiple-choice, we ensure objective, reproducible evaluations without reliance on subjective human or LLM-based judgments. Through comprehensive benchmarking of closed-source and open-source models, we showcase: (i) current MLLMs lack robust audio-visual integration ability and (ii) performance on DeafTest (Pearson’s r = 0.945) strongly correlates with AV-Odyssey accuracy. These findings challenge assumptions about models’ multimodal proficiency and highlight fundamental audio perception as a reasoning bottleneck. We believe that our results provide concrete guidance for future dataset design, alignment strategies, and architectures.