Current evaluation metrics for automated audio captioning (AAC) rely on reference-based comparison, implicitly assuming that a limited reference set adequately represents valid descriptions. However, the inherent cross-modal gap between audio and language gives rise to a many-to-many correspondence: a single audio event can be accurately described by numerous semantically equivalent yet lexically diverse captions. We first demonstrate the resulting instability through ground-truth intra-evaluation on Clotho and AudioCaps, evaluating each human caption against the remaining four references using a leave-one-out protocol. To provide a controlled demonstration, we use a CVAE-based AAC model that generates diverse candidates replicating the ground-truth captions, showing that n-gram metrics reward surface-level replication. We then conduct non-word replacement, synonym replacement, and adjacent-token swapping experiments on both datasets. Overlap metrics penalize synonym replacements almost as strongly as non-word corruption, whereas embedding and CLAP-based metrics respond weakly to local word-order changes. Additional evaluation of CAF-Score and FLEUR (AF3) shows partial progress, but neither configuration clearly separates synonym replacement from adjacent swapping. No evaluated metric jointly provides acoustic grounding, tolerance to valid lexical variation, and local word-order sensitivity. We therefore contribute a reusable AAC metric diagnostic protocol for exposing these trade-offs and guiding future metric development.

This paper presents Emosical, a multimodal open-source dataset of musical films. Emosical comprises video, vocal audio, text, and character identity paired samples with annotated emotion tags. Emosical provides rich emotion annotations for each sample by inferring the background story of the characters. To achieve this, we leverage the musical theatre script, which contains the characters’ complete background stories and narrative contexts. The annotation pipeline includes feeding the speaking character, text, global persona, and context of the dialogue and song track into a large language model. To verify the effectiveness of our tagging scheme, we perform an ablation study by bypassing each step of the pipeline. The ablation results show the usefulness of each component in generating accurate emotion tags. A subjective test is conducted to compare the generated tags of each ablation result. We also perform a statistical analysis to find out the global characteristics of the collected emotion tags. Emosical would enable expressive synthesis and tagging of the speech and singing voice in the musical theatre domain in future research. Emosical is publicly available at https://github.com/gillosae/emosical.