Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion

Yixin Cao, Xiang Ji, Xin Lv, Juanzi Li, Yonggang Wen, Hanwang Zhang


Abstract
We present InferWiki, a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, assumptions, and patterns. First, each testing sample is predictable with supportive data in the training set. To ensure it, we propose to utilize rule-guided train/test generation, instead of conventional random split. Second, InferWiki initiates the evaluation following the open-world assumption and improves the inferential difficulty of the closed-world assumption, by providing manually annotated negative and unknown triples. Third, we include various inference patterns (e.g., reasoning path length and types) for comprehensive evaluation. In experiments, we curate two settings of InferWiki varying in sizes and structures, and apply the construction process on CoDEx as comparative datasets. The results and empirical analyses demonstrate the necessity and high-quality of InferWiki. Nevertheless, the performance gap among various inferential assumptions and patterns presents the difficulty and inspires future research direction. Our datasets can be found in https://github.com/TaoMiner/inferwiki.
Anthology ID:
2021.acl-long.534
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Editors:
Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6855–6865
Language:
URL:
https://aclanthology.org/2021.acl-long.534
DOI:
10.18653/v1/2021.acl-long.534
Bibkey:
Cite (ACL):
Yixin Cao, Xiang Ji, Xin Lv, Juanzi Li, Yonggang Wen, and Hanwang Zhang. 2021. Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6855–6865, Online. Association for Computational Linguistics.
Cite (Informal):
Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion (Cao et al., ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.534.pdf
Optional supplementary material:
 2021.acl-long.534.OptionalSupplementaryMaterial.pdf
Video:
 https://aclanthology.org/2021.acl-long.534.mp4
Code
 TaoMiner/inferwiki
Data
InferWikiFB15k-237