Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction

Samuel Broscheit, Kiril Gashteovski, Yanjie Wang, Rainer Gemulla


Abstract
Open Information Extraction systems extract (“subject text”, “relation text”, “object text”) triples from raw text. Some triples are textual versions of facts, i.e., non-canonicalized mentions of entities and relations. In this paper, we investigate whether it is possible to infer new facts directly from the open knowledge graph without any canonicalization or any supervision from curated knowledge. For this purpose, we propose the open link prediction task,i.e., predicting test facts by completing (“subject text”, “relation text”, ?) questions. An evaluation in such a setup raises the question if a correct prediction is actually a new fact that was induced by reasoning over the open knowledge graph or if it can be trivially explained. For example, facts can appear in different paraphrased textual variants, which can lead to test leakage. To this end, we propose an evaluation protocol and a methodology for creating the open link prediction benchmark OlpBench. We performed experiments with a prototypical knowledge graph embedding model for openlink prediction. While the task is very challenging, our results suggests that it is possible to predict genuinely new facts, which can not be trivially explained.
Anthology ID:
2020.acl-main.209
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2296–2308
Language:
URL:
https://aclanthology.org/2020.acl-main.209
DOI:
10.18653/v1/2020.acl-main.209
Bibkey:
Cite (ACL):
Samuel Broscheit, Kiril Gashteovski, Yanjie Wang, and Rainer Gemulla. 2020. Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2296–2308, Online. Association for Computational Linguistics.
Cite (Informal):
Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction (Broscheit et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.209.pdf
Video:
 http://slideslive.com/38929433
Code
 samuelbroscheit/open_knowledge_graph_embeddings
Data
OLPBENCH