Investigating the Emergent Audio Classification Ability of ASR Foundation Models

Rao Ma, Adian Liusie, Mark Gales, Kate Knill


Abstract
Text and vision foundation models can perform many tasks in a zero-shot setting, a desirable property that enables these systems to be applied in general and low-resource settings. There has been far less work, however, on the zero-shot abilities of ASR foundation models, with these systems typically fine-tuned to specific tasks or constrained to applications that match their training criterion and data annotation. In this work we investigate the ability of Whisper and MMS, ASR foundation models trained primarily for speech recognition, to perform zero-shot audio classification. We use simple template-based text prompts at the decoder and use the resulting decoding probabilities to generate zero-shot predictions. Without training the model on extra data or adding any new parameters, we demonstrate that Whisper shows promising zero-shot classification performance on a range of 8 audio-classification datasets, outperforming the accuracy of existing state-of-the-art zero-shot baselines by an average of 9%. One important step to unlock the emergent ability is debiasing, where a simple unsupervised reweighting method of the class probabilities yields consistent significant performance gains. We further show that performance increases with model size, implying that as ASR foundation models scale up, they may exhibit improved zero-shot performance.
Anthology ID:
2024.naacl-long.266
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4746–4760
Language:
URL:
https://aclanthology.org/2024.naacl-long.266
DOI:
Bibkey:
Cite (ACL):
Rao Ma, Adian Liusie, Mark Gales, and Kate Knill. 2024. Investigating the Emergent Audio Classification Ability of ASR Foundation Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4746–4760, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Investigating the Emergent Audio Classification Ability of ASR Foundation Models (Ma et al., NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.266.pdf
Copyright:
 2024.naacl-long.266.copyright.pdf