On-the-fly Definition Augmentation of LLMs for Biomedical NER

Monica Munnangi, Sergey Feldman, Byron Wallace, Silvio Amir, Tom Hope, Aakanksha Naik


Abstract
Despite their general capabilities, LLMs still struggle on biomedicalNER tasks, which are difficult due to the presence of specialized terminology and lack of training data. In this work we set out to improve LLM performance on biomedical NER in limited data settings via a new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly. During this process, to provide a test bed for knowledge augmentation, we perform a comprehensive exploration of prompting strategies. Our experiments show that definition augmentation is useful for both open source and closed LLMs.For example, it leads to a relative improvement of 15% (on average) in GPT-4 performance (F1) across all (six) of our test datasets. We conduct extensive ablations and analyses to demonstrate that our performance improvements stem from adding relevant definitional knowledge. We find that careful prompting strategies also improve LLM performance, allowing them to outperform fine-tuned language models in few-shot settings. To facilitate future research in this direction, we release our code at https://github.com/allenai/beacon.
Anthology ID:
2024.naacl-long.212
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3833–3854
Language:
URL:
https://aclanthology.org/2024.naacl-long.212
DOI:
Bibkey:
Cite (ACL):
Monica Munnangi, Sergey Feldman, Byron Wallace, Silvio Amir, Tom Hope, and Aakanksha Naik. 2024. On-the-fly Definition Augmentation of LLMs for Biomedical NER. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3833–3854, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
On-the-fly Definition Augmentation of LLMs for Biomedical NER (Munnangi et al., NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.212.pdf
Copyright:
 2024.naacl-long.212.copyright.pdf