Learn to Not Link: Exploring NIL Prediction in Entity Linking

Entity linking models have achieved significant success via utilizing pretrained language models to capture semantic features. However, the NIL prediction problem, which aims to identify mentions without a corresponding entity in the knowledge base, has received insufficient attention. We categorize mentions linking to NIL into Missing Entity and Non-Entity Phrase, and propose an entity linking dataset NEL that focuses on the NIL prediction problem. NEL takes ambiguous entities as seeds, collects relevant mention context in the Wikipedia corpus, and ensures the presence of mentions linking to NIL by human annotation and entity masking. We conduct a series of experiments with the widely used bi-encoder and cross-encoder entity linking models, results show that both types of NIL mentions in training data have a significant influence on the accuracy of NIL prediction. Our code and dataset can be accessed at https://github.com/solitaryzero/NIL_EL


Introduction
Entity Linking (EL) aims to map entity mentions in free texts to their corresponding entities in a given Knowledge Base (KB).Entity linking acts as a bridge between unstructured text and structured knowledge, and benefits various downstream tasks like question answering (Luo et al., 2018) and knowledge extraction (Chen et al., 2021).
However, not all entity mentions correspond to a specific entity in the KB that suits the mention context (Ling et al., 2015).Take Table 1 as an example, Peter Blackburn is actually a journalist, and the householder is a common phrase rather than an entity.These two mentions do not refer to any entity in the given KB.The identification of these mentions is referred to as NIL prediction.Therefore, to tackle NIL prediction, the entity linking model needs to select mentions whose references are absent in KB, and link them to a special placeholder NIL.Dredze et al. (2010) states NIL prediction as one of the key issues in entity linking, which may lead to a decrease in the recall of entity linking systems.Meanwhile, the incorrectly linked entities may provide false information to downstream tasks.
There have been some earlier representation learning based researches that take NIL prediction into consideration (Eshel et al., 2017;Lazic et al., 2015;Peters et al., 2019).They identify NIL mentions by setting a vector similarity threshold or viewing NIL as a special entity.Recently, pretrained language model (PLM) based models (Wu et al., 2020;Fitzgerald et al., 2021;Cao et al., 2021) have achieved great success for their great transferability and expandability.However, these models generally assume that there always exists a correct entity for each mention in the knowledge base, which leaves the NIL prediction problem without adequate attention.
Previous entity linking datasets have paid insufficient attention to the NIL prediction problem.For example, some of the previous datasets like AIDA (Hoffart et al., 2011) view it as an auxiliary task, while others like MSNBC (Cucerzan, 2007) and WNED-WIKI (Eshel et al., 2017) does not require NIL prediction at all.There does not yet exist a strong benchmark for the ability on NIL prediction.
In this paper, we propose an entity linking dataset NEL that focuses on the NIL prediction problem.About 30% of the mentions in NEL do not have their corresponding entity in the candidate entity set, which requires models to identify these mentions rather than linking them to the wrong candidates.In NEL construction, we take ambiguous entities as seeds, and build the dataset by mining mention contexts related to seed entities on the Wikipedia corpus.Then, human annotators are Peter Grant Blackburn (born 25 March 1968) is an Australian badminton player who affiliated with the Ballarat Badminton Association.

Mention Context
Most Hindus accept that there is a duty to have a family during the householder stage of life, as debt to family lineage called Pitra Rin (Father's Debt) and so are unlikely to avoid having children altogether . . .

The Householder (Film)
The Householder (Hindi title: Gharbar) is a 1963 film by Merchant Ivory Productions, with a screenplay by Ruth Prawer Jhabvala . . .The Householder (Novel) The Householder is a 1960 English-language novel by Ruth Prawer Jhabvala . . .asked to identify whether the mentions correspond to a candidate entity or not, and we further perform entity masking to ensure a fair proportion of NIL data of about 30%.
In NIL prediction, we propose to use the widely used bi-encoder and cross-encoder structures as the model backbone, and further integrate type information by adding an entity typing subtask.We combine semantic and type similarity as the final similarity score, and identify NIL mentions by setting a similarity threshold.
We conduct a series of experiments on both NEL and previous entity linking datasets.Experimental results show that the models suffer from an accuracy drop when taking NIL prediction into consideration, indicating that the accuracy may be inflated without the NIL prediction task, and NEL could better diagnose the performance of different models.We also conducted ablation studies on how type information and NIL examples affect the models.We discover that the entity typing subtask yields better embedding even when type similarity is not used, and both types of NIL examples in training data would boost the ability of NIL prediction.
Our contributions can be concluded as: • We categorize the NIL prediction problem into two patterns: Missing Entity and Non-Entity Phrase, where the latter one has not received sufficient attention in previous works.
• We propose an entity linking dataset NEL focusing on NIL prediction, which covers two patterns of NIL data and could act as a benchmark for diagnosing the ability of NIL prediction.
• We conducted a series of experiments, whose results demonstrate that the accuracy of models may be inflated when not taking NIL prediction into consideration.Meanwhile, both patterns of NIL data in training are essential for triggering the ability of NIL prediction.

Preliminary
Entity mentions M = {m i } refer to text spans potentially corresponding to entities in a document D = (w 1 , w 2 , ..., w n ), where w i is either a plain token or a mention.Each mention m i may correspond to an entity e i ∈ E in the entity set E of knowledge base KB.
Definition 1. Entity linking aims to find an optimal mapping Γ : M ⇒ E, which maps entity mentions M = {m i } to their corresponding entities E = {e i }, where e i ∈ KB.
The NIL prediction problem is to determine whether an entity mention m is absent from the knowledge base KB.When there does not exist a proper entity e ∈ KB for the given mention m, As demonstrated in Table 1, there exist two situations in real-world entity linking where NIL prediction should be taken into consideration: • Missing Entity: The mention m refers to certain entity e that has not been yet included in KB, i.e. m ⇒ e / ∈ KB.For example, in the upper half of Table 1, the mention Peter Blackburn refers to a certain journalist, while entries in English Wikipedia include only people with other occupations, leading to the mention linking to NIL.
• Non-Entity Phrase: The mention m refers to a common phrase that is not usually viewed as an entity, i.e. m ⇏ e.For example, the mention the householder in the lower half of Table 1 refers to a concept rather than a film or novel.

Dataset Construction
There does not yet exist a strong benchmark on NIL prediction.We manually annotated 300 examples with their mentions linking to NIL from the widely used entity linking dataset AIDA 1 , and discover that about 10% of these mentions should actually link to an entity.For example, the mention "EU" in "EU rejects German" should be linked to the European Union rather than NIL (See Appendix D for details).Meanwhile, NIL mentions in AIDA fall mostly in the Missing Entity category.The incorrect and imbalanced data for NIL prediction indicates that the importance of NIL prediction is currently underestimated.
In this section, we propose an entity linking dataset named NEL, which focuses on the NIL prediction problem.The construction process is demonstrated in Figure 1.
Unlike normal entity linking data, there does not exist annotated references for mentions linking to NIL, and the portion of NIL data in the text corpus is unknown.Hyperlinks in Wikipedia can be viewed as mentions linking to non-NIL entities, from which we can find the aliases of entities.We assume that when an alias does not appear as a hyperlink in a certain context, it may be identified as a mention linking to NIL.In this way, we collect such contexts as the raw data.The raw data is then annotated by humans to ensure correctness, and we further mask out some answer entities in the candidate set to control the percentage of NIL in answers.

Data Collection
Levin (1977) states that the title of creative works could be a place, a personal name, or a certain abstract concept like the choric embodiment of some collectivity (The Clouds, The Birds) or stock types (The Alchemist, Le Misanthrope), which would naturally lead to the two situations where a mention links to NIL.The absence of the referenced entity from the KB would lead to Missing Entity, while an abstract concept not viewed as an entity would lead to Non-Entity Phrase.
To discover NIL mentions of both types, we start by taking entities that share an alias with other entities as seeds.We assume that a mention referring to multiple entities has a higher probability of linking to a Missing Entity outside the KB, and the complex meaning of the mention will lead to Non-Entity Phrase.Thus, the aliases of ambiguous seed entities would be good starting points for mining NIL mentions.

Entity Selection
We further filter ambiguous entities to remove low-quality seeds.First, we remove noise instances like template pages, and entities with less than 5 hyperlinks are also removed.Meanwhile, we discarded entities with a probability greater than 50% of being the linking result, as these entities can generally be considered to be unambiguous and lack difficulty.Finally, 1000 entities are sampled as the seed entity set E s .
We use a typing system based on Wikidata to identify the type of selected entities.We view the instance of relation as the type indicator, and utilize the subclass of relation to build a tree-form type system.The detailed type system can be found in Appendix C.

Mention Discovery
We build an alias table from the 2021 Wikipedia dump by extracting alias-entity pairs (m, e) from internal hyperlinks.All alias m related to entities in the seed entity set E s are gathered as the mention set M .For each mention m ∈ M , we look for its occurrence throughout the Wikipedia corpus (whether it appears as a hyperlink or not) to obtain the entry tuple (C l , m, C r , E m ), where C l and C r represent contexts left and right to the entity mention m, and E m represents the candidate entities set of m.For each mention m, we sampled 5 entries where m appears as a hyperlink and 5 entries where m appears in plain text to balance the number of positive and negative examples, and a total of 10,000 entries are collected.

Human Annotation and Post-processing
We perform annotation on the above entries with 3 annotators.The annotators are provided with the mention context (C l , m, C r ) and candidate entities E m .Each candidate entity e consists of its title, textual description, and Wikipedia URL.The annotators are asked to choose the answer entity a ∈ E m corresponding to the mention m, or a = N IL if none of the candidates are correct.
An expert will further investigate entries in which annotators fail to reach a consensus.The expert is a senior annotator with essential knowledge of entity linking, and will confirm the final annotation result after carefully reading through the context and candidate entities.We use the annotation result as the final answer a if there is an agreement between 3 annotators, and the entity chosen by the expert otherwise.The annotated tuple (C l , m, C r , E m , a) acts as the final entry of our dataset.
To further simulate the situation where new emerging entities do not appear in knowledge bases, we perform entity masking on positive entries.We randomly sample 10% entries where a ̸ = N IL, and mask the correct entity in the candidate set E m .In this case, as the correct answer is removed from the candidate list, we have a = N IL, i.e. the mention m corresponds to the empty entity NIL.
A person is invited to bless the child.
Bless the Child is a 2000 supernatural horror film.
Bless the Child (Film) "Bless the Child" is the seventh single by Nightwish.

Bless the Child (Song)
Hotel Blessing The Blessing (song) Blessing, Texas

Type Space
Activity Song Film

Bi-Encoder
A person is invited to bless the child.
Bless the Child is a 2000 supernatural horror film.
A person is invited to bless the child.
"Bless the Child" is the seventh single by Nightwish.

Threshold
Figure 2: The overall structure of PLM-based retrieval models.Candidates which are confusing in the semantic space may be more distinguishable in the type space.Mentions linking to NIL frequently differ from their candidates in their types, so we combine semantic similarity with type similarity for NIL prediction.

Entity Linking with NIL prediction
A mention links to NIL when all of its candidate entities fail to match its context.A common paradigm of NIL prediction is to compute the similarity scores between the mention context and candidates, and judge its linking result on the base of similarities.

Scoring Similarities
Bi-encoder and cross-encoder are widely adopted scorer structures, as they are well compatible with pretrained language models.Bi-encoder encodes mention contexts and entities into the same dense vector space, while cross-encoder views similarity scoring as a sequence classification task: where f, g, h are PLM-based encoders, W and b are trainable variables, and σ refers to the sigmoid function.
The bi-encoder structure allows precomputing entity embeddings in knowledge bases, which enables efficient retrieval in real-world applications.Compared with bi-encoder, cross-encoder better captures the relation between context and entities with the cross attention mechanism, thus demonstrating higher accuracy.

Integrating Entity Types
Previous entity linking models (Gupta et al., 2017;Onoe and Durrett, 2020;Raiman and Raiman, 2018) have proved that entity types do help models better disambiguate between candidate entities.
The type information can be integrated into biencoders and cross-encoders by adding a typing layer.In the bi-encoder structure, the mention types  To tackle the label imbalance between types, we use the focal loss (Lin et al., 2017) on the typing task: ) where n t is the total number of types in the type system, γ is a hyperparameter, y i is the golden label of the i-th type and t i is the predicted label of the i-th type.In bi-encoder, L t is the average of loss on t c and t e , while in cross-encoder L t is directly calculated from [t c , t e ].
We train the semantic encoder with binary classification loss L s , and combine L s with L t as the final loss L:

Identifying Mentions Linking to NIL
The type similarity is computed with cosine similarity, and the final score is a weighted sum between type similarity and semantic similarity: where λ is a hyperparameter.For each entry (C l , m, C r , E m , a), we concatenate (C l , m, C r ) to form the context c.In the training step, for each candidate entity e ∈ E m , we view (c, e) as a positive example if e = a, and as a negative example if a = N IL or e ̸ = a.
During evaluation, the similarity score s(c, e) is computed between context c and each candidate entity e.If there exist entities with a score equal to or higher than the nil threshold ϵ = 0.5, we choose the entity with the highest similarity score as the answer; If all entities fail to reach the threshold, then the mention m links to NIL.

Experiments
We conduct a series of experiments on two types of datasets, which test the different ability of entity linking models: (1) NEL that tests the ability of NIL prediction; (2) previous EL datasets that tests the ability of entity disambiguation.We choose the following models for comparison: BLINK (Wu et al., 2020) that uses the bi-encoder and crossencoder alone to score candidate entities, CLINK that integrates type information with BLINK, and GENRE (Cao et al., 2021) that generates the linking result with a sequence-to-sequence model.

Main Experiment Results
We trained and tested the models on NEL, with BERT-large as the encoder base of BLINK and CLINK, and BART-large as the backbone of GENRE, to observe their ability in NIL prediction.We also experimented on previous entity linking datasets to observe the disambiguation ability of different models.The models are trained on the AIDA-YAGO2-train (Hoffart et al., 2011) dataset, and tested on AIDA-YAGO2-testb, MSNBC (Cucerzan, 2007) and WNED-WIKI (Eshel et al., 2017).187 distinct types are used in experiments on NEL, and considering that the entity type distribution may be different across datasets, we use a type system with only 14 top-level entity types on previous datasets to make CLINK more transferable (See Appendix C for details).We retain the same textual representation format with BLINK (see Appendix A), while using 128 as the length of context sequences and entity descriptions.All models are implemented with PyTorch and optimized with the Adam (Kingma and Ba, 2014) optimizer with a learning rate of 1e-5.
Table 3 shows the evaluation results, from which some insights could be discovered: Type Information Matters.CLINK with crossencoder structure achieves the highest accuracy on almost all datasets, and is still comparable with GENRE on AIDA, from which we may assert that taking type information into consideration is helpful even without NIL prediction.Meanwhile, on both structures, the overall accuracy of CLINK outperforms BLINK on all datasets, proving that the entity typing task assists both bi-encoder and cross-encoder distinguish correct entities.
Encoder Structure.The cross-encoder structure generally performs better than bi-encoder, but we observe that sometimes bi-encoders show de-cent ability in detecting NIL mentions, especially when type information is not utilized.BLINKbi achieves the highest NIL accuracy score of 69.59 on AIDA with NIL, and has a score of 88.59 on NEL, which is comparable with the bestperforming CLINK-cross.This phenomenon indicates that cross-encoders may be more prone to overfitting, while entity types would alleviate this tendency.
NIL Entries.On the AIDA dataset, we observe that the models generally suffer from a drop in accuracy when taking NIL entries into consideration, and the drop is more obvious in bi-encoders.This may indicate that the performance of models is inflated without NIL prediction, and NIL entries may confuse the models in practical application.

Influence of the Entity Typing Task
We conducted experiments to observe how the entity typing task influences the model.We trained the model with both L t and L s as loss, while setting λ = 1 during evaluation to ignore the influence of type similarity scores.Results shown in Table 4 reflects two observations: First, candidate type predictions benefit from mention context.We observe that when changing the structure from bi-encoder to cross-encoder, the type prediction accuracy on candidates raises by 5%, where the accuracy on contexts remains unchanged.This is likely because the context helps narrow down the type range, while the context type prediction generally remains unaffected by entity descriptions.
Second, unifying the entity typing task in the training process leads to higher overall accuracy, even when the type similarity score is not taken into consideration in the evaluation, which can be demonstrated by the improved score of OAC w/ Typing compared to OAC w/o Typing on both models.This may indicate that predicting entity types would help the model learn semantic embeddings with higher quality.

Influence of NIL Training Data
Compared with previous datasets like AIDA, NEL contains more NIL mentions of the Non-Entity Phrase type.We trained the models with different numbers of Non-Entity Phrase examples to observe the influence of NIL Training Data.
As demonstrated by Figure 3, all models suffer from a great decline in NIL accuracy when no NIL examples are used during the training stage, and biencoder is more prone to the accuracy drop.However, by using only 25% of Non-Entity Phrase examples in training, the NIL accuracy would recover to a decent level.Further adding NIL examples has little impact on cross-encoder models, but bi-encoder models still constantly benefit from additional data.
Besides, ignoring NIL data with the Non-Entity Phrase type will also harm the NIL accuracy and overall accuracy.Both types of NIL training data are necessary to reach to best performance.
We discover that entity linking models may be unaware of the NIL mentions when there is insufficient training data.A small amount of training data is enough for cross-encoder models to reach a reasonable accuracy, while bi-encoder models constantly benefit from additional training data.

Related Work
PLM-based Models in Entity Linking.Using pretrained language models (PLM) to capture semantic information is widely adopted in recent entity linking models.BLINK (Wu et al., 2020) marks the mention in context with pre-defined special tokens, and takes BERT as the encoder base.Two structures are adopted by BLINK to handle different situations: bi-encoder for fast dense retrieval, and cross-encoder for further disambiguation.MOLEMAN (Fitzgerald et al., 2021) searches for similar mention contexts instead of entities, which better captures the diverse aspects an entity reflects in various contexts.GENRE (Cao et al., 2021) finetunes the sequence-to-sequence model BART, directly generating the unique entity name according to the mention context.
Research on NIL Prediction.The NIL prediction problem has been long viewed as an auxiliary task of entity linking.Some entity linking datasets (AIDA (Hoffart et al., 2011), TAC-KBP series (Mc-Namee and Dang, 2009)) take the NIL prediction problem into consideration, while some (ACE and MSNBC) (Ratinov et al., 2011) omit mentions linking to NIL.Some research has already been conducted on the NIL prediction problem.Lazic et al. (2015) and Peters et al. ( 2019) set a score threshold to filter reasonable candidates, and mentions with no candidate score above the threshold are linked to NIL.Sil and Florian (2016); Kolitsas et al. (2018) views the NIL placeholder as a special entity, and selecting it as the best match indicates that the mention refers to no entities in the given KB.However, recent entity linking models, which use pretrained language models (PLM) as encoder bases, generally take the in-KB setting, which assumes that each mention has a valid golden entity in the KB (Wu et al., 2020).
Entity Type Assisted Entity Linking.Entity types can effectively assist entity linking and have been studied in various works.Gupta et al. (2017) jointly encodes mention context, entity description, and Freebase types with bidirectional LSTM to maximize the cosine similarity.DeepType (Raiman and Raiman, 2018) predicts the type probability of each token and gathers relevant tokens to predict the entity types, which would help eliminate candidates with incompatible types.Onoe and Durrett (2020) views entity types as a training objective rather than a feature, predicting fine-grained Wikipedia category tags to select the most relevant entity.

Conclusion
In this paper, we propose an entity linking dataset NEL that focuses on the NIL prediction problem.We observe that mentions linking to NIL can be categorized into two patterns: Missing Entity and Non-Entity Phrase, but the latter one has not been paid sufficient attention.We propose an entity linking dataset NEL that focuses on NIL prediction.The dataset is built upon the Wikipedia corpus by choosing ambiguous entities as seeds and collecting relevant mention contexts.NEL is humanannotated to ensure correctness, and entity masking is further performed to control the percentage of NIL.
We conducted a series of experiments to examine the performance of PLM-based models on different datasets.Experimental results indicate that the accuracy without considering NIL prediction would be inflated.Meanwhile, sufficient data of both NIL types during training is essential to trigger the ability of NIL prediction.In the future, we may further try to integrate entity types into the pretraining process and explore type transfer between datasets.

Limitations
Our work still exist some limitations.First, we choose an entity typing system on the base of Wikidata tags, however, the granularity of the typing system remains to be discussed.A system with too many types would introduce noise to long-tail types, while insufficient types would weaken the disambiguation ability of type similarity.Thus, building a type system with adequate granularity remains a challenge.
Second, we combine the entity typing task with PLM-based semantic encoders, which require a fixed type system and further finetuning.Integrating the entity typing task into the pretraining process may enhance the transferability of the model and remove the dependency on a fixed type system.Potential Risks.Our proposed dataset NEL centers on ambiguous entities, whose type distribution may not remain the same with other datasets.A potential risk is that the model trained on NEL may experience under-exposure of other entity types, which would damage their transferability and lead to undesired outputs on other datasets.

Ethics Statement
In this section, we will discuss the ethical considerations of our work.
Licenses and terms.The Wikipedia corpus and Wikidata types are obtained via the Wikimedia dump2 , under the CC BY-SA 3.0 license3 .AIDA, MSNBC, and WNED-WIKI are shared under the CC BY-SA 3.0 license.These datasets have been widely used in entity linking research, and we believe that they have been anonymized and desensitized.
Human Annotation.We recruited 3 human annotators without a background of expertise in annotation, and 1 expert annotator with adequate knowledge in entity linking for checking.These annotators are employed by commercial data annotation companies.We have paid these recruited annotators with adequate rewards under the agreed working time and price.The annotators are well informed about how these annotated data will be used and released, which has been recorded in the contract.
Intended use.NEL is an entity linking dataset focusing on the NIL prediction problem.Researchers are intended to use NEL for examining the ability of NIL prediction of newly created entity linking models.AIDA, MSNBC, and WNED-WIKI are intended for entity linking research, which is compatible with our work.

E Case Study
Table 8 shows some examples predicted by CLINK and BLINK without type information, which reflects how entity types influence the linking result.
In the first example, models with the bi-encoder structure incorrectly take the "Gates of Heaven" entry (which is in fact a documentary film) as the linking result, while CLINK-cross notices the context word "album" may indicate the entity type type, and correctly links the mention to the album.In the second example, the "Home Before Dark" mention actually refer to a 1997 movie4 like other movies in the context, however the corresponding entry is absent in the English Wikipedia.The CLINK-cross model is able to identify that the mention should be labelled as NIL, where the other models mistakenly link it to the "Home Before Dark" entry, which is an album rather than a movie.We observe that the entity types do help measure the similarity

t
e = σ(W e g(e) + b e ) (4) while they are simultaneously predicted in the cross-encoder structure: [t c , t e ] = σ(W f ([c, e]) + b) (5) where σ represents the sigmoid function and W c , b c , W e , b e , W, b are trainable parameters.

Figure 3 :
Figure 3: Ablation study on the influence of NIL training data.The x-axis indicates the percentage of used NIL data or Non-Entity Phrase data in the training set.

Table 1 :
Example of mentions that should be linked to NIL and their potential candidate entities.Mentions are labeled as red.

Table 2 :
Statistics of the NEL dataset compared with previous entity linking datasets.*The percentage of two NIL patterns in AIDA is calculated from 300 randomly sampled NIL data, and data with errors do not count as any pattern.

Table 2
Compared with previous entity linking datasets, NEL has a higher percentage of NIL data, which could better diagnose the ability of different models in NIL prediction.Meanwhile, mentions linking to NIL in AIDA mostly fall in the Missing Entity situation, while NEL focuses more on the Non-Entity Phrase situation, thus complementing the insufficient attention on Non-Entity Phrase in NIL prediction.

Table 3 :
(Cao et al., 2021)ts on NEL and previous datasets.Non-NAC, NAC, and OAC represent non-NIL accuracy, NIL accuracy and overall accuracy.*Results of GENRE on AIDA w/o NIL, MSNBC, and WNED-WIKI are taken from the original paper(Cao et al., 2021).

Table 4 :
Experimental results on the influence of the entity typing task on NEL.OAC indicates the overall accuracy of entity linking.The overall accuracy with typing is achieved without using the type similarity score.
ModelCtxt Type Acc.Cand Type Acc.OAC w/ Typing OAC w/o Typing

Table 6 :
Examples of type lines Wikidata types.Table6shows some examples of type lines in the system.The most 10 frequent types in NEL are: (Work, Organisation, Place, Event, Person, Activity, FictionalCharacter, Award, Species, MeanOfTransportation).C.2 Typing System on Traditional DatasetsWe retain 14 top-level types to make CLINK more transferable on different datasets.These types are: (Other, Person, Place, Work, Organization, Event, Fictional Character, Species, Activity, Device, Topical Concept, Ethnic Group, Food, Disease) D Errors in AIDA Table 7 demonstrates some mentions in AIDA that are incorrectly linked to NIL.50 errors are detected among the 300 randomly sampled data in AIDA.