Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods

Juhyuk Lee, Min-Joong Lee, June Yong Yang, Eunho Yang


Abstract
The ability to extract entities and their relations from unstructured text is essential for the automated maintenance of large-scale knowledge graphs. To keep a knowledge graph up-to-date, an extractor needs not only the ability to recall the triples it encountered during training, but also the ability to extract the new triples from the context that it has never seen before. In this paper, we show that although existing extraction models are able to easily memorize and recall already seen triples, they cannot generalize effectively for unseen triples. This alarming observation was previously unknown due to the composition of the test sets of the go-to benchmark datasets, which turns out to contain only 2% unseen data, rendering them incapable to measure the generalization performance. To separately measure the generalization performance from the memorization performance, we emphasize unseen data by rearranging datasets, sifting out training instances, or augmenting test sets. In addition to that, we present a simple yet effective augmentation technique to promote generalization of existing extraction models, and experimentally confirm that the proposed method can significantly increase the generalization performance of existing models.
Anthology ID:
2022.naacl-main.282
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3849–3858
Language:
URL:
https://aclanthology.org/2022.naacl-main.282
DOI:
10.18653/v1/2022.naacl-main.282
Bibkey:
Cite (ACL):
Juhyuk Lee, Min-Joong Lee, June Yong Yang, and Eunho Yang. 2022. Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3849–3858, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods (Lee et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.282.pdf
Video:
 https://aclanthology.org/2022.naacl-main.282.mp4
Data
WebNLG