Mental health, defined by psychological, emotional, and social well-being, has emerged as a significant social issue in modern society. To assess an individual’s mental state and provide personalized support, clinician–patient interviews are commonly conducted. However, given the constraints on access to such interviews, conversational agents have the potential to provide more accessible mental health support. In this paper, we present , a multimodal conversational agent that assesses an individual’s emotional and depressive states using an audio–text model that jointly models linguistic and paralinguistic cues from speech, and provides personalized mental health support through LLM-based response generation. Quantitative and qualitative evaluations indicate improvements in multimodal state assessment and response quality over the considered baselines.

Multimodal Emotion Recognition in Conversation aims to identify emotions within a dialogue with multimodal data, including audio, visual, and textual features. While existing methods have made significant improvements, there are two fundamental limitations to be addressed. From the modality fusion perspective, current approaches treat all modalities as functionally equivalent during fusion, overlooking their distinct communicative roles and information capacities, in which text conveys explicit semantic meaning while audio provides paralinguistic cues. From the emotion label perspective, many works ignore the continuous structure of emotion characterized by psychological theory and apply uniform penalties regardless of affective proximity. To address these limitations, we propose EMART, EMotion-Wheel-Guided Audio-Referred Text Representation for ERC, specifically focusing on audio and text modalities. First, we propose a modality-aware fusion strategy capturing linguistic features from text as the primary source and audio as a complementary component. Secondly, we propose an emotion-wheel-guided supervised contrastive loss to encode emotional proximity based on Russell’s circumplex model. Experimental results on IEMOCAP and MELD demonstrate outstanding performance. The code is available at: https://github.com/DILAB-HYU/EMART.git.