2025
pdf
bib
abs
“Stupid robot, I want to speak to a human!” User Frustration Detection in Task-Oriented Dialog Systems
Mireia Hernandez Caralt
|
Ivan Sekulic
|
Filip Carevic
|
Nghia Khau
|
Diana Nicoleta Popa
|
Bruna Guedes
|
Victor Guimaraes
|
Zeyu Yang
|
Andre Manso
|
Meghana Reddy
|
Paolo Rosso
|
Roland Mathis
Proceedings of the 31st International Conference on Computational Linguistics: Industry Track
Detecting user frustration in modern-day task-oriented dialog (TOD) systems is imperative for maintaining overall user satisfaction, engagement, and retention. However, most recent research is focused on sentiment and emotion detection in academic settings, thus failing to fully encapsulate implications of real-world user data. To mitigate this gap, in this work, we focus on user frustration in a deployed TOD system, assessing the feasibility of out-of-the-box solutions for user frustration detection. Specifically, we compare the performance of our deployed keyword-based approach, open-source approaches to sentiment analysis, dialog breakdown detection methods, and emerging in-context learning LLM-based detection. Our analysis highlights the limitations of open-source methods for real-world frustration detection, while demonstrating the superior performance of the LLM-based approach, achieving a 16% relative improvement in F1 score on an internal benchmark. Finally, we analyze advantages and limitations of our methods and provide an insight into user frustration detection task for industry practitioners.
2024
pdf
bib
abs
Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems
Ivan Sekulic
|
Silvia Terragni
|
Victor Guimarães
|
Nghia Khau
|
Bruna Guedes
|
Modestas Filipavicius
|
Andre Ferreira Manso
|
Roland Mathis
Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024)
In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator’s coherence with user goals, effectively mitigating hallucinations—a major source of inconsistencies in simulator responses.
2015
pdf
bib
Boosting Named Entity Recognition with Neural Character Embeddings
Cícero dos Santos
|
Victor Guimarães
Proceedings of the Fifth Named Entity Workshop