Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation

Jarana Manotumruksa, Jeff Dalton, Edgar Meij, Emine Yilmaz


Abstract
While state-of-the-art Dialogue State Tracking (DST) models show promising results, all of them rely on a traditional cross-entropy loss function during the training process, which may not be optimal for improving the joint goal accuracy. Although several approaches recently propose augmenting the training set by copying user utterances and replacing the real slot values with other possible or even similar values, they are not effective at improving the performance of existing DST models. To address these challenges, we propose a Turn-based Loss Function (TLF) that penalises the model if it inaccurately predicts a slot value at the early turns more so than in later turns in order to improve joint goal accuracy. We also propose a simple but effective Sequential Data Augmentation (SDA) algorithm to generate more complex user utterances and system responses to effectively train existing DST models. Experimental results on two standard DST benchmark collections demonstrate that our proposed TLF and SDA techniques significantly improve the effectiveness of the state-of-the-art DST model by approximately 7-8% relative reduction in error and achieves a new state-of-the-art joint goal accuracy with 59.50 and 54.90 on MultiWOZ2.1 and MultiWOZ2.2, respectively.
Anthology ID:
2021.findings-emnlp.144
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1674–1683
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.144
DOI:
10.18653/v1/2021.findings-emnlp.144
Bibkey:
Cite (ACL):
Jarana Manotumruksa, Jeff Dalton, Edgar Meij, and Emine Yilmaz. 2021. Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1674–1683, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation (Manotumruksa et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.144.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.144.mp4
Code
 feay1234/tlf-sda
Data
DialoGLUE