D2U: Distance-to-Uniform Learning for Out-of-Scope Detection

Eyup Yilmaz, Cagri Toraman


Abstract
Supervised training with cross-entropy loss implicitly forces models to produce probability distributions that follow a discrete delta distribution. Model predictions in test time are expected to be similar to delta distributions if the classifier determines the class of an input correctly. However, the shape of the predicted probability distribution can become similar to the uniform distribution when the model cannot infer properly. We exploit this observation for detecting out-of-scope (OOS) utterances in conversational systems. Specifically, we propose a zero-shot post-processing step, called Distance-to-Uniform (D2U), exploiting not only the classification confidence score, but the shape of the entire output distribution. We later combine it with a learning procedure that uses D2U for loss calculation in the supervised setup. We conduct experiments using six publicly available datasets. Experimental results show that the performance of OOS detection is improved with our post-processing when there is no OOS training data, as well as with D2U learning procedure when OOS training data is available.
Anthology ID:
2022.naacl-main.152
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2093–2108
Language:
URL:
https://aclanthology.org/2022.naacl-main.152
DOI:
10.18653/v1/2022.naacl-main.152
Bibkey:
Cite (ACL):
Eyup Yilmaz and Cagri Toraman. 2022. D2U: Distance-to-Uniform Learning for Out-of-Scope Detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2093–2108, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
D2U: Distance-to-Uniform Learning for Out-of-Scope Detection (Yilmaz & Toraman, NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.152.pdf
Video:
 https://aclanthology.org/2022.naacl-main.152.mp4
Data
HWU64SNIPS