COMBO: A Complete Benchmark for Open KG Canonicalization

Chengyue Jiang, Yong Jiang, Weiqi Wu, Yuting Zheng, Pengjun Xie, Kewei Tu


Abstract
Open knowledge graph (KG) consists of (subject, relation, object) triples extracted from millions of raw text. The subject and object noun phrases and the relation in open KG have severe redundancy and ambiguity and need to be canonicalized. Existing datasets for open KG canonicalization only provide gold entity-level canonicalization for noun phrases. In this paper, we present COMBO, a Complete Benchmark for Open KG canonicalization. Compared with existing datasets, we additionally provide gold canonicalization for relation phrases, gold ontology-level canonicalization for noun phrases, as well as source sentences from which triples are extracted. We also propose metrics for evaluating each type of canonicalization. On the COMBO dataset, we empirically compare previously proposed canonicalization methods as well as a few simple baseline methods based on pretrained language models. We find that properly encoding the phrases in a triple using pretrained language models results in better relation canonicalization and ontology-level canonicalization of the noun phrase. We release our dataset, baselines, and evaluation scripts at path/to/url.
Anthology ID:
2023.eacl-main.26
Volume:
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
340–357
Language:
URL:
https://aclanthology.org/2023.eacl-main.26
DOI:
10.18653/v1/2023.eacl-main.26
Bibkey:
Cite (ACL):
Chengyue Jiang, Yong Jiang, Weiqi Wu, Yuting Zheng, Pengjun Xie, and Kewei Tu. 2023. COMBO: A Complete Benchmark for Open KG Canonicalization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 340–357, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
COMBO: A Complete Benchmark for Open KG Canonicalization (Jiang et al., EACL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eacl-main.26.pdf
Video:
 https://aclanthology.org/2023.eacl-main.26.mp4