Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach

Yuepei Li, Kang Zhou, Qiao Qiao, Qing Wang, Qi Li


Abstract
Distantly-Supervised Named Entity Recognition (DS-NER) uses knowledge bases or dictionaries for annotations, reducing manual efforts but rely on large human labeled validation set. In this paper, we introduce a real-life DS-NER dataset, QTL, where the training data is annotated using domain dictionaries and the test data is annotated by domain experts. This dataset has a small validation set, reflecting real-life scenarios. Existing DS-NER approaches fail when applied to QTL, which motivate us to re-examine existing DS-NER approaches. We found that many of them rely on large validation sets and some used test set for tuning inappropriately. To solve this issue, we proposed a new approach, token-level Curriculum-based Positive-Unlabeled Learning (CuPUL), which uses curriculum learning to order training samples from easy to hard. This method stabilizes training, making it robust and effective on small validation sets. CuPUL also addresses false negative issues using the Positive-Unlabeled learning paradigm, demonstrating improved performance in real-life applications.
Anthology ID:
2025.coling-main.727
Volume:
Proceedings of the 31st International Conference on Computational Linguistics
Month:
January
Year:
2025
Address:
Abu Dhabi, UAE
Editors:
Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10940–10959
Language:
URL:
https://aclanthology.org/2025.coling-main.727/
DOI:
Bibkey:
Cite (ACL):
Yuepei Li, Kang Zhou, Qiao Qiao, Qing Wang, and Qi Li. 2025. Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10940–10959, Abu Dhabi, UAE. Association for Computational Linguistics.
Cite (Informal):
Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach (Li et al., COLING 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.coling-main.727.pdf