Knowledge Base Question Answering through Recursive Hypergraphs

Knowledge Base Question Answering (KBQA) is the problem of predicting an answer for a factoid question over a given knowledge base (KB). Answering questions typically requires reasoning over multiple links in the given KB. Humans tend to answer questions by grouping different objects to perform reasoning over acquired knowledge. Hypergraphs provide a natural tool to model group relationships. In this work, inspired by typical human intelligence, we propose a new method for KBQA based on hypergraphs. Existing methods for KBQA, though effective, do not explicitly incorporate the recursive relational group structure in the given KB. Our method, which we name RecHyperNet (Recursive Hypergraph Network), exploits a new way of modelling KBs through recursive hypergraphs to organise such group relationships in KBs. Experiments on multiple KBQA benchmarks demonstrate the effectiveness of the proposed RecHyperNet. We have released the code.


Introduction
Knowledge Base Question Answering (KBQA) (Xu et al., 2016)), a task that tests the ability of a machine to understand knowledge like a human, is a challenging, central, and popular task in natural language processing. KBQA is the problem of predicting an answer for a factoid question over a given knowledge base (KB) containing facts such as (Inception, written by, Christopher Nolan). Answering questions typically requires reasoning over multiple links in the given KB (Zhang et al., 2018).
When a typical human answers a question (e.g. What are the genres of movies written by Christopher Nolan?), they tend to group objects (e.g. movies written by Christopher Nolan such as Inception, Interstellar, etc. are grouped together) over their acquired knowledge. Hypergraphs are mathematical tools that naturally encode group relationships. Directed hypergraphs have been recently used to model KBs for KBQA (Han et al., 2020) but we argue that they fail to naturally model group relationships without loss of information. Spcifically, two KB triples such as (Inception, written by, Christopher Nolan) and (Person of Interest, written by, Jonathan Nolan) would be modelled by a directed hyperedge {Inception, Person of Interest} -> {Christopher Nolan, Jonathan Nolan} clearly resulting in loss of information (not clear who directed which). Though objects of similar types are grouped together and directions are encoded, directed hypergraphs fail to model KBs without loss of information.
We argue that recursive hypergraphs (Joslyn and Nowak, 2017; Menezes and Roth, 2019) provide a flexible way to model KBs without loss of information. Specifically, all movies written by Christopher Nolan are grouped together and those written by Jonathan Nolan are separately grouped together. To summarise, we make the following contributions • We model KBs as recursive hypergraphs. To the best of our knowledge, this is the first such attempt for natural language processing.
• We propose RecHyperNet (Recursive Hypergraph Network), a novel graph neural networkbased model to exploit recursive hypergraphs and apply it for KBQA.
• We show the effectiveness of RecHyperNet on multiple KBQA benchmarks. We have released the source code in the supplementary.

Related Work
Knowledge Base Question Answering (KBQA) has been widely investigated especially through end-to-end deep neural networks since the release of popular datasets such as MetaQA (Zhang et al., 2018). One of the first attempts at KBQA was key-value memory network (KVMN) (Miller et al., 2016) that maintained memory to store KB facts and text as key-value pairs. Graphs of Relations Among Facts and Text Networks (Graft-Net) (Sun et al., 2018) (Marcheggiani and Titov, 2017), and neural machine translation (Bastings et al., 2017). A recent tutorial touches upon the recent advances of GNNs in NLP (Vashishth et al., 2019).
GNNs for Question Answering (QA): A popular task closely related to question answering is multi-hop reasoning across documents (context passages) where GNNs have been extensively used (Qiu et al., 2019b;Ding et al., 2019;Tu et al., 2020). Graph Attention Networks (Veličković et al., 2018) have been shown to be effective for modelling the multi-grained structure of documents for machine reading comprehension (MRC) (Zheng et al., 2020). GNNs have also recently been used on heterogeneous data sources for MRC (Kim et al., 2019;Tu et al., 2019), knowledge graphs for QA (De Cao et al., 2019;Cao et al., 2019;Lin et al., 2019;Qiu et al., 2019a;Ji et al., 2020;Shao et al., 2020), answer sentence selection (Tian et al., 2020), and QA on tables (Zhang, 2020) and math (Chen et al., 2020). In all these publications except 2HR-DR (Han et al., 2020), the input is a graph and limited to modelling pairwise relationships. We address this fundamental limitation by modelling a knowledge graph as a recursive hypergraph 2HR-DR models knowledge graphs through directed hypergraphs but we argue that they are more naturally represented by recursive hypergraphs.

Method
We describe the KBQA problem, and how recursive hypergraphs (Menezes and Roth, 2019; Joslyn and Nowak, 2017) can be used to model knowledge bases. We then describe our method of exploiting recursive hypergraphs for KBQA.

KBQA Problem
The KBQA problem considered in this work is as follows. We are given a knowledge base K ⊆ E × R × E with entity set E and relation set R. We are also given a natural language question q with a topic entity e ∈ E. The task is to output an answer a ∈ E that correctly answers the question q.

Recursive Hypergraph
In this subsection, we precisely define a recursive hypergraph by first defining the following.
Definition 1 (Depth k Powerset). For a set S, let us use S(S) to denote the powerset of S i.e. S(S) := {Ṡ : ∅ ⊆Ṡ ⊆ S}. Then, the the depth k powerset of S is Note that a hypergraph in the traditional sense is a 0-recursive hypergraph. We call a hyperedge e ∈ E as a depth k hyperedge if e ⊆ 2 V,k but e 2 V,k .

Modelling knowledge Base as a Recursive Hypergraph
One of our main contributions in this work is to model a knowledge base as a 1-recursive hypergraph with V = E ∪ R as the set of vertices. Each head entity can be seen as a depth 1 hyperedge connecting all its relations. Each relation can be further seen as a depth 0 hyperedge connecting all the tail entities. For example, if a knowledge base contains the movie "Inception" as a head entity with (relation, object) pairs as 1) (starred actors, Leonardo DiCaprio), 2) (starred actors, Ellen Page), 3) (starred actors, Tom Hardy), 4) (genre, action), 5) (genre, adventure), and 6) (genre, science fiction), and 7) (written by, Christopher Nolan) then we can view them as a recursive hypergraph with Inception as a depth 1 hyperedge connecting relation vertices starred actors, genre, and written by. Each of these relation vertices is in turn a depth 0 hyperedge connecting object entities. For example, the object entities Leonardo DiCaprio, Ellen Page, and Tom Hardy are contained in a single hyperedge (that represents the pair Inception, starred actors).

RecHyperNet for KBQA
In this section, we describe our proposed methodology for KBQA. The main ingredients/modules of RecHyperNet are three modules: KB Embedding Module, Topic and Question Embedding Module, and Answer Retrieval Module. 1) KB Embedding Module uses knowledge base embedding methods to initialise entity embeddings of the input KB. We denote these initial embeddings by x e , e ∈ E. Popular knowledge base embedding methods include TransE (Bordes et al., 2013), and ComplEX (Trouillon et al., 2016).
2) Topic and Question Embedding Module embeds the question q to a fixed dimension vector q of dimension d. We use a feed-forward neural network that first represents q using LSTM/ RoBERTa (Liu et al., 2019). The topic entity e that is also present in the question is embedded using a multirelational graph convolutional network on the KB (Vashishth et al., 2020). The entity update rule for Composition-based multi-relational graph convolutional network (CompGCN) is as follows: where x u and z r are initial features of vertex u and relation r respectively. φ is a composition function that is dictated by the knowledge base embedding method and f is a non-linear activation function such as Rectified Linear Unit. N (v) is the (relation, object) neighbourhood of the vertex v (for example, the 7 pairs listed for "inception" movie entity in the previous subsection).
Our key modification to CompGCN update rule is motivated through the proposed recursive hypergraph view of knowledge base. Specifically, since each relation can be seen as a hyperedge containing all object entities, we can modify the update rule as follows where N (v) is the set of all relations in the neighbourhood of v, and N (v, r) is the set of all object entities connected by the (subject, relation) pair (v, r). max is the element-wise maximum of a set of vectors. We can replace max by other aggregator functions such as mean, and sum but we experimentally observed that max gives the best performance. We embed the topic head entity e ∈ E using Equation 3 and obtain the hidden representation h e with the same dimension d. We finally pass the concatenated representation h e ||q to a 2-layer feed-forward neural network to get the topic and question representation.

3) Answer Selection Module Given a KB embedding scoring function Φ and a set of answer entities
A ⊆ E, we learn the topic and question representation h e ||q so that Φ(x e , h e ||q, x a ) > 0, ∀a ∈ A and Φ(x e , h e ||q, xā) > 0, ∀ā / ∈ A. We leartn the model parameters by minimising the binary cross entropy loss between the sigmoid of the scores and the target labels (1 for correct, 0 for wrong).
During test time, we score the given (topic head entity, question) pair against all possible answers a ∈ E. So, the answer is given by arg max a ∈E Φ(x e , h e ||q, x a ).

Experiments
We evaluate our proposed method on MetaQA (Zhang et al., 2018) and WebQuestionsSP (Yih et al., 2016) datasets. We closely follow the experimental setup of a prior work (Saxena et al., 2020) for the preprocessed versions of these datasets. MetaQA consists of 400k questions (1 − hop, 2hop, and 3-hop), and a knowledge graph of 135k triples, 43k entities, and 9 relations. WebQuestion-sSP consists of 4700 questions and a knowledge  graph of 1.8 million entities, and 5.7 million triples. Following prior work (Saxena et al., 2020), we experimented on two different settings (for both datasets) -KG Full (in which the KG is left untouched), and the more realistic KG-50 setting in which 50% links are randomly removed. We compared against 7 different baselines as shown in Table 1. Please see Section 2 (subsection: Knowledge Base Question Answering) for brief descriptions of the baseline methods.
Model details Following prior work (Saxena et al., 2020), we used a long short term memory (LSTM) network to learn embeddings for words in the questions with an embedding size of 256 for MetaQA and RoBERTa (768 dimensional embeddings) (Liu et al., 2019) for WebQuestionsSP datasets. The hidden dimension size for graph convolutional network was also set to 256. A dropout rate of 0.2 was used for all neural layers.. All models were implemented in PyTorch (Paszke et al., 2019) and trained with ADAM as the optimiser (Kingma and Ba, 2015), a learning rate of 5×10 −4 , weight decay of 1.0, a batch size of 128 trained for 100 epochs (with patience of 5).
Results. Table 1 shows the results. We evaluated our proposed method and all baselines through Hits@1 metric. As we can see in Table 1  Net utilise additional text corpus (unrealistic setting as it is not always readily available) to answer questions while our method does not. We exploit only the given knowledge base structure. Futhermore, as shown in Table 2, baselines such as GraftNet and PullNet perform poorly in the absence of text.

Ablation Analysis
We conducted an ablation study by removing essential components from RecHyperNet. Specifically we replaced our proposed Equation 3 to exploit recursive hypergraphs by the exisdting Equation 2 of CompGCN. As another basline, we removed the embedding of topic head entity obtained through GCN and used only the question representation while scoring. As shown in Table 3, both these components are essential for our proposed RecHyperNet.

Conclusion and Future Work
In this work, we exploit recursive hypergraphs in NLP for the task of KBQA. We have proposed a novel method based on graph convolutional networks. We have demonstrated the effectiveness of RecHyperNet on KBQA benchmarks. In future, we exploit recursive structures for other tasks where graph neural nets are effective such as question generation (Pan et al., 2020), sentiment analysis , etc.