Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition

Floris Hengst, Ralf Wolter, Patrick Altmeyer, Arda Kaygan


Abstract
We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level.By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection.In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection.CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.
Anthology ID:
2024.findings-naacl.156
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2412–2432
Language:
URL:
https://aclanthology.org/2024.findings-naacl.156
DOI:
Bibkey:
Cite (ACL):
Floris Hengst, Ralf Wolter, Patrick Altmeyer, and Arda Kaygan. 2024. Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2412–2432, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition (Hengst et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.156.pdf
Copyright:
 2024.findings-naacl.156.copyright.pdf