Large Audio Language Models (LALMs) are increasingly applied in affective and conversational AI, yet their ability to align with human empathetic judgments in spoken dialogue remains poorly understood. We introduce BAEmpEval, a multimodal evaluation framework for analyzing empathetic response judgment in instruction-following LALMs under subjective conversational settings. The framework contains 5,436 dialogue pairs collected from three speech sources, where models evaluate listener responses across six fine-grained dimensions spanning both vocal perception (Vocal Warmth, Perceived Genuineness, Prosodic Attunement) and conversational response strategies (Exploration, Interpretation, Emotional Reaction). Experiments reveal a challenging setting: experiments across recent LALMs reveal uneven alignment with human relational judgments across empathy dimensions and speech conditions. Our findings suggest that empathetic response judgment remains a challenging multidimensional perceptual task for current LALMs and highlight the importance of evaluation frameworks that account for subjective relational variability. We release the dataset and evaluation code to support future research in human-centric AI.

In the context of today’s high-pressure, aging society, the demand for large-scale emotional models capable of providing empathetic support is more critical than ever. However, existing benchmarks fail to simultaneously achieve ecological validity, signal clarity, and reliable fine-grained labeling. We introduce EmoS, a high-fidelity bilingual benchmark designed to resolve the limitations of ecological validity and noise in existing datasets by combining strictly filtered static slices with a dynamic Streaming Monologue subset. Supported by a rigorous dual-layer human annotation pipeline, EmoS provides trusted ground truth that captures continuous emotional evolution. Empirical results show that fine-tuning MLLMs (multimodal large language models) on EmoS yields significant gains over zero-shot baselines, laying the foundation for the training and evaluation of future emotion recognition models and empathy models. The dataset and code are publicly available at https://github.com/NLP2CT/EmoS.

Large Audio Language Models (LALMs) exhibit strong speech understanding ability, yet it remains unclear whether they meaningfully utilize subtle speech-conditioned emotional cues beyond coarse acted emotions. In this work, we investigate whether the evaluated LALMs exhibit measurable sensitivity to fine-grained emotional variation in speech through controlled perturbation experiments, fine-grained emotion elicitation tasks, and representation geometry analysis. Our results show that vocal delivery variations systematically influence both model behavior and hidden representations, indicating that the evaluated LALMs already encode non-trivial speech-conditioned affective information. However, models struggle under emotionally ambiguous settings, particularly for semantically related fine-grained emotions. Representation analysis further suggests that subtle emotional structure is partially preserved despite weak downstream utilization. Overall, our findings suggest that the evaluated LALMs do not completely ignore subtle speech emotion cues, but instead utilize them inconsistently under ambiguous affective conditions