Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop

Md Tahmid Rahman Laskar, Cheng Chen, Xue-yong Fu, Shashi Bhushan Tn


Abstract
Telephone transcription data can be very noisy due to speech recognition errors, disfluencies, etc. Not only that annotating such data is very challenging for the annotators, but also such data may have lots of annotation errors even after the annotation job is completed, resulting in a very poor model performance. In this paper, we present an active learning framework that leverages human in the loop learning to identify data samples from the annotated dataset for re-annotation that are more likely to contain annotation errors. In this way, we largely reduce the need for data re-annotation for the whole dataset. We conduct extensive experiments with our proposed approach for Named Entity Recognition and observe that by re-annotating only about 6% training instances out of the whole dataset, the F1 score for a certain entity type can be significantly improved by about 25%.
Anthology ID:
2022.dash-1.12
Volume:
Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances)
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates (Hybrid)
Editors:
Eduard Dragut, Yunyao Li, Lucian Popa, Slobodan Vucetic, Shashank Srivastava
Venue:
DaSH
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
88–93
Language:
URL:
https://aclanthology.org/2022.dash-1.12
DOI:
Bibkey:
Cite (ACL):
Md Tahmid Rahman Laskar, Cheng Chen, Xue-yong Fu, and Shashi Bhushan Tn. 2022. Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop. In Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances), pages 88–93, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Cite (Informal):
Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop (Laskar et al., DaSH 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.dash-1.12.pdf
Video:
 https://aclanthology.org/2022.dash-1.12.mp4