Investigating the Effect of Background Knowledge on Natural Questions

Vidhisha Balachandran, Bhuwan Dhingra, Haitian Sun, Michael Collins, William Cohen


Abstract
Existing work shows the benefits of integrating KBs with textual evidence for QA only on questions that are answerable by KBs alone (Sun et al., 2019). In contrast, real world QA systems often have to deal with questions that might not be directly answerable by KBs. Here, we investigate the effect of integrating background knowledge from KBs for the Natural Questions (NQ) task. We create a subset of the NQ data, Factual Questions (FQ), where the questions have evidence in the KB in the form of paths that link question entities to answer entities but still must be answered using text, to facilitate further research into KB integration methods. We propose and analyze a simple, model-agnostic approach for incorporating KB paths into text-based QA systems and establish a strong upper bound on FQ for our method using an oracle retriever. We show that several variants of Personalized PageRank based fact retrievers lead to a low recall of answer entities and consequently fail to improve QA performance. Our results suggest that fact retrieval is a bottleneck for integrating KBs into real world QA datasets
Anthology ID:
2021.deelio-1.3
Volume:
Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
Month:
June
Year:
2021
Address:
Online
Venue:
DeeLIO
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
25–30
Language:
URL:
https://aclanthology.org/2021.deelio-1.3
DOI:
10.18653/v1/2021.deelio-1.3
Bibkey:
Cite (ACL):
Vidhisha Balachandran, Bhuwan Dhingra, Haitian Sun, Michael Collins, and William Cohen. 2021. Investigating the Effect of Background Knowledge on Natural Questions. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 25–30, Online. Association for Computational Linguistics.
Cite (Informal):
Investigating the Effect of Background Knowledge on Natural Questions (Balachandran et al., DeeLIO 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.deelio-1.3.pdf
Data
Natural Questions