A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries

Elena Epure, Romain Hennequin


Abstract
We conducted a human subject study of named entity recognition on a noisy corpus of conversational music recommendation queries, with many irregular and novel named entities. We evaluated the human NER linguistic behaviour in these challenging conditions and compared it with the most common NER systems nowadays, fine-tuned transformers. Our goal was to learn about the task to guide the design of better evaluation methods and NER algorithms. The results showed that NER in our context was quite hard for both human and algorithms under a strict evaluation schema; humans had higher precision, while the model higher recall because of entity exposure especially during pre-training; and entity types had different error patterns (e.g. frequent typing errors for artists). The released corpus goes beyond predefined frames of interaction and can support future work in conversational music recommendation.
Anthology ID:
2023.eacl-main.92
Volume:
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1281–1296
Language:
URL:
https://aclanthology.org/2023.eacl-main.92
DOI:
10.18653/v1/2023.eacl-main.92
Award:
 EACL Outstanding Paper
Bibkey:
Cite (ACL):
Elena Epure and Romain Hennequin. 2023. A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1281–1296, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries (Epure & Hennequin, EACL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eacl-main.92.pdf
Video:
 https://aclanthology.org/2023.eacl-main.92.mp4