Davan Harrison
2024
Active Listening: Personalized Question Generation in Open-Domain Social Conversation with User Model Based Prompting
Kevin Bowden
|
Yue Fan
|
Winson Chen
|
Wen Cui
|
Davan Harrison
|
Xin Wang
|
Marilyn Walker
Findings of the Association for Computational Linguistics: EMNLP 2024
Large language models (LLMs) capable of casual conversation have recently become widely available. We hypothesize that users of conversational systems want a more personalized experience, and existing work shows that users are highly receptive to personalized questions (PQs). Question Generation tasks, however, focus on factual questions from textual excerpts. To create a PQ generator, we first identify over 400 real user interests by anonymously aggregating ~39K user models. We then populate prompt templates with these 400 interests and use an LLM to generate PQs customized to user interests. The result is PerQs, a novel corpus of ~19K question/answer pairs. We evaluate PerQs at scale in the unique context of the Alexa Prize. Our results show significant positive effects on perceived conversation quality. We then fine-tune, deploy, and evaluate PerQy, a neural model that generates PQs in real-time. When evaluated against several competitive LLM baselines, PerQy produced the most natural and engaging responses.
Search
Co-authors
- Kevin Bowden 1
- Yue Fan 1
- Winson Chen 1
- Wen Cui 1
- Xin Wang 1
- show all...