Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process audio–video data remains underexplored. This gap highlights the need for a high-quality benchmark to systematically evaluate MLLM performance in a real-world setting.

We introduce SONIC-O1, a comprehensive, fully human-verified benchmark of ≈60 hours (231 clips) spanning 13 real-world conversational domains with 4,958 annotations and perceived demographic metadata. SONIC-O1 evaluates three capabilities: open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (reasoning). Across closed- and open-source models, we find that the MCQ accuracy shows the smallest gap between model families, under their recommended settings, the best closed-source model outperforms the best open-source model by 22.6% on temporal localization.

We further observe accuracy gaps of up to 21.4% on temporal localization across demographic groups, indicating diagnostic disparities in model behaviour. SONIC-O1 provides an open evaluation suite for temporally grounded and demographically informed multimodal understanding.

We release SONIC-O1 for reproducibility and research: Project Page: https://vectorinstitute.github.io/sonic-o1, Dataset: https://huggingface.co/datasets/vector-institute/sonic-o1 , Code Repository: https://github.com/VectorInstitute/sonic-o1.

Building Agent Assistants that can help improve customer service support requires inputs from industry users and their customers, as well as knowledge about state-of-the-art Natural Language Processing (NLP) technology. We combine expertise from academia and industry to bridge the gap and build task/domain-specific Neural Agent Assistants (NAA) with three high-level components for: (1) Intent Identification, (2) Context Retrieval, and (3) Response Generation. In this paper, we outline the pipeline of the NAA’s core system and also present three case studies in which three industry partners successfully adapt the framework to find solutions to their unique challenges. Our findings suggest that a collaborative process is instrumental in spurring the development of emerging NLP models for Conversational AI tasks in industry. The full reference implementation code and results are available at https://github.com/VectorInstitute/NAA.