LLM4RE: A Data-centric Feasibility Study for Relation Extraction

Anushka Swarup, Tianyu Pan, Ronald Wilson, Avanti Bhandarkar, Damon Woodard


Abstract
Relation Extraction (RE) is a multi-task process that is a crucial part of all information extraction pipelines. With the introduction of the generative language models, Large Language Models (LLMs) have showcased significant performance boosts for complex natural language processing and understanding tasks. Recent research in RE has also started incorporating these advanced machines in their pipelines. However, the full extent of the LLM’s potential for extracting relations remains unknown. Consequently, this study aims to conduct the first feasibility analysis to explore the viability of LLMs for RE by investigating their robustness to various complex RE scenarios stemming from data-specific characteristics. By conducting an exhaustive analysis of five state-of-the-art LLMs backed by more than 2100 experiments, this study posits that LLMs are not robust enough to tackle complex data characteristics for RE, and additional research efforts focusing on investigating their behaviors at extracting relationships are needed. The source code for the evaluation pipeline can be found at https://aaig.ece.ufl.edu/projects/relation-extraction .
Anthology ID:
2025.coling-main.447
Volume:
Proceedings of the 31st International Conference on Computational Linguistics
Month:
January
Year:
2025
Address:
Abu Dhabi, UAE
Editors:
Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6670–6691
Language:
URL:
https://aclanthology.org/2025.coling-main.447/
DOI:
Bibkey:
Cite (ACL):
Anushka Swarup, Tianyu Pan, Ronald Wilson, Avanti Bhandarkar, and Damon Woodard. 2025. LLM4RE: A Data-centric Feasibility Study for Relation Extraction. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6670–6691, Abu Dhabi, UAE. Association for Computational Linguistics.
Cite (Informal):
LLM4RE: A Data-centric Feasibility Study for Relation Extraction (Swarup et al., COLING 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.coling-main.447.pdf