Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process audio–video data remains underexplored. This gap highlights the need for a high-quality benchmark to systematically evaluate MLLM performance in a real-world setting.

We introduce SONIC-O1, a comprehensive, fully human-verified benchmark of ≈60 hours (231 clips) spanning 13 real-world conversational domains with 4,958 annotations and perceived demographic metadata. SONIC-O1 evaluates three capabilities: open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (reasoning). Across closed- and open-source models, we find that the MCQ accuracy shows the smallest gap between model families, under their recommended settings, the best closed-source model outperforms the best open-source model by 22.6% on temporal localization.

We further observe accuracy gaps of up to 21.4% on temporal localization across demographic groups, indicating diagnostic disparities in model behaviour. SONIC-O1 provides an open evaluation suite for temporally grounded and demographically informed multimodal understanding.

We release SONIC-O1 for reproducibility and research: Project Page: https://vectorinstitute.github.io/sonic-o1, Dataset: https://huggingface.co/datasets/vector-institute/sonic-o1 , Code Repository: https://github.com/VectorInstitute/sonic-o1.

Language models increasingly serve as advisory systems in maintenance operations. To prevent hallucination, established approaches ground these models in procedural documentation, constraining them to prescribed sequences. In practice, however, operators may stray from these steps, requiring models to recognise unscripted, off-procedure utterances. Current benchmarks rarely prioritise this capability. We introduce DiagFlowBench, a dataset of 50 industrial diagnostic flowcharts from a consumer manufacturer converted into 1,676 multi-turn conversations that contrast compliant with off-procedure utterances. Evaluating a panel of ten commercial and open-weight models reveals high variability in abstention rates, with models often selecting a real but contextually inadequate step rather than fabricating facts. The inherent plausibility and authority of this mapped but wrong advice exposes a challenging vulnerability for grounding systems.