Multi-Criteria Evaluation Framework of Selecting Response-worthy Chats in Live Streaming

Zhantao Lai, Kosuke Sato


Abstract
Live streaming, a dynamic medium that merges real-time audiovisual content with interactive text-based chat, presents unique challenges for maintaining viewer engagement and ensuring streamers’ well-being. This study introduces a multi-criteria evaluation framework designed to identify response-worthy chats during live streaming. We proposed a system that evaluates chats based on sentiment polarity and intensity, contextual relevance, and topic uniqueness. We also constructed a dataset annotated by human reviewers who validates the framework, demonstrating a closer alignment with human preferences compared to single-criterion baselines. This framework not only supports the development of more responsive and engaging live streaming environments but also contributes to the broader field of dialog systems by highlighting the distinct needs of real-time, large-scale conversational contexts.
Anthology ID:
2024.sigdial-1.16
Volume:
Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Month:
September
Year:
2024
Address:
Kyoto, Japan
Editors:
Tatsuya Kawahara, Vera Demberg, Stefan Ultes, Koji Inoue, Shikib Mehri, David Howcroft, Kazunori Komatani
Venue:
SIGDIAL
SIG:
SIGDIAL
Publisher:
Association for Computational Linguistics
Note:
Pages:
186–191
Language:
URL:
https://aclanthology.org/2024.sigdial-1.16
DOI:
10.18653/v1/2024.sigdial-1.16
Bibkey:
Cite (ACL):
Zhantao Lai and Kosuke Sato. 2024. Multi-Criteria Evaluation Framework of Selecting Response-worthy Chats in Live Streaming. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 186–191, Kyoto, Japan. Association for Computational Linguistics.
Cite (Informal):
Multi-Criteria Evaluation Framework of Selecting Response-worthy Chats in Live Streaming (Lai & Sato, SIGDIAL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.sigdial-1.16.pdf