Assistants on assembly tasks show great potential to benefit humans ranging from helping with everyday tasks to interacting in industrial settings. However, evaluation resources in assembly activities are underexplored. To foster system development, we propose a new multimodal QA evaluation dataset on assembly activities. Our dataset, ProMQA-Assembly, consists of 646 QA pairs that require multimodal understanding of human activity videos and their instruction manuals in an online-style manner. For cost effectiveness in the data creation, we adopt a semi-automated QA annotation approach, where LLMs generate candidate QA pairs and humans verify them. We further improve QA generation by integrating fine-grained action labels to diversify question types. Additionally, we create 81 instruction task graphs for our target assembly tasks. These newly created task graphs are used in our benchmarking experiment, as well as in facilitating the human verification process. With our dataset, we benchmark models, including competitive proprietary multimodal models. We find that ProMQA-Assembly contains challenging multimodal questions, where reasoning models showcase promising results. We believe our new evaluation dataset contributes to the further development of procedural-activity assistants.

We present VDAct 2.0, an enhanced benchmark for video-grounded dialogue that builds upon the original VDAct by expanding dialogue coverage and introducing a scalable LLM-assisted filtering pipeline to ensure high-quality, grounded QA pairs. VDAct 2.0 comprises 6,356 human-annotated dialogues with a total of 63,958 turns, grounded in 2,975 household activity videos, with undesirable dialogue turns systematically identified and removed. To achieve this, we design a trigger-based quality framework and calibrate a panel of high-agreement LLMs through human-in-the-loop calibration, allowing scalable QA-turn-level filtering. We benchmark a wide range of pretrained and fine-tuned models, both open-source and proprietary, across standard text generation metrics and LLM-based evaluations. The results highlight both recent advances and remaining challenges in video-grounded dialogue modeling, positioning VDAct 2.0 as a high-fidelity testbed for evaluating and advancing multimodal reasoning in interactive settings.

Effective task-oriented dialogue (TOD) requires agents to coordinate two tightly coupled decisions of when to invoke external tools and how to formulate a response that satisfies the user’s goal. However, their turn-level behavior remains weakly grounded under fixed zero-shot prompts. Meanwhile, existing retrieval-based TOD agents primarily retrieve task-relevant content from external knowledge sources, guiding which information to include but offering limited guidance on how to select tools and formulate responses in similar dialogue contexts. To mitigate this, we introduce TRACE, a Turn-level Retrieval Agent with Contextual Experiences for TOD [<https://github.com/aistairc/TRACE-TOD>]. Instead of relying on retrieved factual records alone, TRACE retrieves goal-conditioned dialogue snippets as behavioral demonstrations at each turn. These snippets capture similar user goals, constraints, tool-use decisions, and response patterns to guide both tool calls and response generation. Experiments on MultiWOZ 2.1 and Schema-Guided Dialogue (SGD) show that TRACE improves Inform and Success over a competitive LLM agent and end-to-end fine-tuned models under both simulation-based and static-user evaluations. Human evaluation and retrieval-focused analyses further show that goal-conditioned snippets and retrieval quality drive these gains.

Video-grounded dialogue (VGD) requires models to generate coherent responses by tracking and reasoning over temporally evolving events in video content. However, existing approaches offer limited ability to capture temporal dependencies, particularly in complex scenarios. In this paper, we present a framework that integrates structured event timelines with verification-aware reasoning to improve temporal consistency in VGD. We represent video content as structured states and temporally ordered actions, enabling models to both infer event timelines from video segments and leverage them to guide response generation. In addition, to address the inherent noise in predicted structure timelines, we introduce a Generate-Verify-Refine (GVR) mechanism that explicitly evaluates and revises unreliable event predictions, thereby reducing the propagation of upstream errors. Experiments on the VDAct benchmark show that our approach consistently improves dialogue quality across models, achieving gains of up to +13.0 LAVE and accuracy over unstructured grounding. Human evaluation further demonstrates improvements of 20.7-31.9 points in response correctness. Additional analysis shows that while better event predictions correlate with stronger dialogue performance, naive reliance on structured outputs can degrade results, highlighting the need for verification-aware reasoning. Our implementation of GVR on event timelines for VGD is available at https://github.com/aistairc/GVR-VGD.

Multimodal systems have great potential to assist humans in procedural activities, where people follow instructions to achieve their goals. Despite diverse application scenarios, systems are typically evaluated on traditional classification tasks, e.g., action recognition or temporal action localization. In this paper, we present a novel evaluation dataset, ProMQA, to measure the advancement of systems in application-oriented scenarios. ProMQA consists of 401 multimodal procedural QA pairs on user recording of procedural activities, i.e., cooking, coupled with their corresponding instruction. For QA annotation, we take a cost-effective human-LLM collaborative approach, where the existing annotation is augmented with LLM-generated QA pairs that are later verified by humans. We then provide the benchmark results to set the baseline performance on ProMQA. Our experiment reveals a significant gap between human performance and that of current systems, including competitive proprietary multimodal models. We hope our dataset sheds light on new aspects of models’ multimodal understanding capabilities.

This paper presents a schema-aware end-to-end neural network model for handling task-oriented dialogues based on a dynamic set of slots within a schema. Contrary to existing studies that proposed end-to-end approaches for task-oriented dialogue systems by relying on a unified schema across domains, we design our approach to support a domain covering multiple services where diverse schemas are available. To enable better generalizability among services and domains with different schemas, we supply the schema’s context information including slot descriptions and value constraints to the model. The experimental results on a well-known Schema-Guided Dialogue (SGD) dataset demonstrated the performance improvement by the proposed model compared to state-of-the-art baselines in terms of end-to-end modeling, dialogue state tracking task, and generalization on new services and domains using a limited number of dialogues.