Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems

Ivan Sekulic, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, Roland Mathis


Abstract
In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator’s coherence with user goals, effectively mitigating hallucinations—a major source of inconsistencies in simulator responses.
Anthology ID:
2024.scichat-1.3
Volume:
Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024)
Month:
March
Year:
2024
Address:
St. Julians, Malta
Editors:
Yvette Graham, Qun Liu, Gerasimos Lampouras, Ignacio Iacobacci, Sinead Madden, Haider Khalid, Rameez Qureshi
Venues:
SCI-CHAT | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
19–35
Language:
URL:
https://aclanthology.org/2024.scichat-1.3
DOI:
Bibkey:
Cite (ACL):
Ivan Sekulic, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, and Roland Mathis. 2024. Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems. In Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024), pages 19–35, St. Julians, Malta. Association for Computational Linguistics.
Cite (Informal):
Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems (Sekulic et al., SCI-CHAT-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.scichat-1.3.pdf
Video:
 https://aclanthology.org/2024.scichat-1.3.mp4