Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization

Yangyang Zhao, Mehdi Dastani, Jinchuan Long, Zhenyu Wang, Shihan Wang


Abstract
Training a task-oriented dialogue policy using deep reinforcement learning is promising but requires extensive environment exploration. The amount of wasted invalid exploration makes policy learning inefficient. In this paper, we define and argue that dead-end states are important reasons for invalid exploration. When a conversation enters a dead-end state, regardless of the actions taken afterward, it will continue in a dead-end trajectory until the agent reaches a termination state or maximum turn. We propose a Dead-end Detection and Resurrection (DDR) method that detects dead-end states in an efficient manner and provides a rescue action to guide and correct the exploration direction. To prevent dialogue policies from repeating errors, DDR also performs dialogue data augmentation by adding relevant experiences that include dead-end states and penalties into the experience pool. We first validate the dead-end detection reliability and then demonstrate the effectiveness and generality of the method across various domains through experiments on four public dialogue datasets.
Anthology ID:
2024.tacl-1.86
Volume:
Transactions of the Association for Computational Linguistics, Volume 12
Month:
Year:
2024
Address:
Cambridge, MA
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
1578–1596
Language:
URL:
https://aclanthology.org/2024.tacl-1.86/
DOI:
10.1162/tacl_a_00717
Bibkey:
Cite (ACL):
Yangyang Zhao, Mehdi Dastani, Jinchuan Long, Zhenyu Wang, and Shihan Wang. 2024. Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization. Transactions of the Association for Computational Linguistics, 12:1578–1596.
Cite (Informal):
Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization (Zhao et al., TACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.tacl-1.86.pdf