AVERT: Audio-Verified Adjudication for Spoken Dialogue State Tracking

Chunggi Lee, Hanspeter Pfister


Abstract
Spoken dialogue state tracking recovers slot-value pairs from speech, where ASR errors concentrate in entity values and persist across turns, making it both a generation and an editing problem. A strong per-turn text editor corrects much of this but, operating on the transcript alone, leaves three recoverable errors: a value predicted inconsistently across turns, an omitted slot, and a value the audio does not support. We present AVERT, which scores each candidate value by combining cross-turn agreement with a trained audio-conditioned verifier and resolves the three error types with three operators, vote, add, and swap, each restricted to the slots where its error is common. On SpokenWOZ, a base speech-LLM reaches 33.04 JGA, a text editor 38.34, and AVERT 40.13, without retraining either. This is in the range of a 1B end-to-end system that consumes the full spoken history (39.32), though AVERT uses two 1B decoders rather than one. The audio verifier contributes a statistically significant gain, and restricting each operator to a selected slot subset matters: removing it lets unrestricted voting overwrite correct categorical values and fall below the editor.
Anthology ID:
2026.emnlp-main.1759
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
40120–40130
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1759/
DOI:
Bibkey:
Cite (ACL):
Chunggi Lee and Hanspeter Pfister. 2026. AVERT: Audio-Verified Adjudication for Spoken Dialogue State Tracking. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 40120–40130, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
AVERT: Audio-Verified Adjudication for Spoken Dialogue State Tracking (Lee & Pfister, EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1759.pdf