Auditing Speaker-Level Metadata Utility in Text-Residual Prosody Prediction

Gaeun Kim, Jihyun Park, Sohee Jang, Joon-Hyuk Chang


Abstract
Text only partially determines prosodic realization, leaving residual variation that may align with speaker-level metadata such as regional accent or first-language (L1) background. However, because these labels are assigned to speakers, their predictive gains may overlap with broader speaker-associated structure rather than uniquely reflect category-level prosodic effects. We frame this as an attribution problem and propose a speaker-controlled audit using matched text-residual prosody predictors that differ only in access to metadata labels and speaker identity. The audit measures label utility beyond text, speaker utility beyond text, and the additional empirical utility of the label when speaker conditioning is already available. Across VCTK regional-accent labels and L2-ARCTIC L1 labels, metadata yields modest gains over text, while speaker conditioning yields larger gains and leaves little additional label utility under speaker control. Random speaker-group and label-perturbation controls further characterize the specificity and use of the supplied metadata. Importantly, speaker-held-out evaluation shows that metadata transfer to unseen speakers is limited and target-dependent, indicating that observed-speaker metadata gains do not by themselves establish speaker-independent category-level effects. These results position speaker-controlled auditing as a prerequisite attribution check before assigning category-level interpretations to metadata-conditioned predictive gains.
Anthology ID:
2026.findings-emnlp.2212
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
47393–47412
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.2212/
DOI:
Bibkey:
Cite (ACL):
Gaeun Kim, Jihyun Park, Sohee Jang, and Joon-Hyuk Chang. 2026. Auditing Speaker-Level Metadata Utility in Text-Residual Prosody Prediction. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 47393–47412, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Auditing Speaker-Level Metadata Utility in Text-Residual Prosody Prediction (Kim et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.2212.pdf