Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems

Zhenpeng Su, Xing W, Wei Zhou, Guangyuan Ma, Songlin Hu


Abstract
Dialogue response selection aims to select an appropriate response from several candidates based on a given user and system utterance history. Most existing works primarily focus on post-training and fine-tuning tailored for cross-encoders. However, there are no post-training methods tailored for dense encoders in dialogue response selection. We argue that when the current language model, based on dense dialogue systems (such as BERT), is employed as a dense encoder, it separately encodes dialogue context and response, leading to a struggle to achieve the alignment of both representations. Thus, we propose Dial-MAE (Dialogue Contextual Masking Auto-Encoder), a straightforward yet effective post-training technique tailored for dense encoders in dialogue response selection. Dial-MAE uses an asymmetric encoder-decoder architecture to compress the dialogue semantics into dense vectors, which achieves better alignment between the features of the dialogue context and response. Our experiments have demonstrated that Dial-MAE is highly effective, achieving state-of-the-art performance on two commonly evaluated benchmarks.
Anthology ID:
2024.naacl-long.47
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
820–830
Language:
URL:
https://aclanthology.org/2024.naacl-long.47
DOI:
Bibkey:
Cite (ACL):
Zhenpeng Su, Xing W, Wei Zhou, Guangyuan Ma, and Songlin Hu. 2024. Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 820–830, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems (Su et al., NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.47.pdf
Copyright:
 2024.naacl-long.47.copyright.pdf