Andrew Li
2024
ChatHF: Collecting Rich Human Feedback from Real-time Conversations
Andrew Li
|
Zhenduo Wang
|
Ethan Mendes
|
Duong Minh Le
|
Wei Xu
|
Alan Ritter
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
We introduce ChatHF, an interactive annotation framework for chatbot evaluation, which integrates configurable annotation within a chat interface. ChatHF can be flexibly configured to accommodate various chatbot evaluation tasks, for example detecting offensive content, identifying incorrect or misleading information in chatbot responses, and chatbot responses that might compromise privacy. It supports post-editing of chatbot outputs and supports visual inputs, in addition to an optional voice interface. ChatHF is suitable for collection and annotation of NLP datasets, and Human-Computer Interaction studies, as demonstrated in case studies on image geolocation and assisting older adults with daily activities. ChatHF is publicly accessible at https://chat-hf.com.
Search