Large Language Models (LLMs) are impactful yet opaque artifacts. Subsymbolic by design, they are defined by billions of numeric weights that interact in a largely inscrutable manner. Current analysis paradigms are either black-box benchmarks that test model performance on pre-defined tasks, or mechanistic interpretability approaches that trace back outputs to specific weights. Both analysis methods are limited by the experimenter’s hypothesis space - one must know what to look for to find it. In this article, we argue for a third, radically different analysis paradigm: Epistemic Twins. We propose constructing large-scale symbolic approximations of LLMs in human-readable formats. This enables the comprehensive inspection of factual knowledge (or beliefs) inherent in the model without predefining hypotheses, facilitating large-scale auditing towards better understanding and explainability.

LLMs are remarkable artifacts that have revolutionized a range of knowledge-intensive tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we provide a framework and the results of a multi-dimensional analysis of GPTKB v1.5 (CITATION), a recursively elicited Knowledge Base (KB) of 100 million facts (or beliefs) of a frontier LLM, namely, GPT-4.1. Given the scale of the elicited facts, we provide a multi-dimensional approach to qualitatively and quantitatively analyze these facts as opposed to the mainstream fact completion benchmarks, which are prone to availability bias. We find that the models’ factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous benchmarks. We also find that inconsistency, ambiguity and hallucinations are major issues, shedding light on future research opportunities in neuro-symbolic AI concerning extraction, consolidation and verification of factual LLM knowledge.

We propose a vision of EdgeKBs(Edge Knowledge Bases) integrating KBs into Edge AI systems. EdgeKBs can infuse commonsense and domain knowledge within Edge devices to guide them while learning from training data. By doing so, we not only bring explainability and performance benefits of KBs to the Edge, but also enable more efficient, sustainable Edge AI systems. We envisage three approaches to develop EdgeKBs: Slicing Dicing; Tailoring to Fit; and Dynamic EdgeKBs. These EdgeKBs will bring Knowledge-Guided Machine Learning (KGML) to the Edge to support next-generation applications, e.g., Physical AI, cobots, and local natural language interfaces.

Music festival lineups emerge from complex relationships among artists, genres, releases, labels, and past performances, making the prediction of future lineups a natural fit for temporal knowledge graph (TKG) forecasting. In this work, we present a TKG covering 380 festivals over 55 years, comprising more than 90K festival performance quadruples along with information on festivals, artist tours, and artist metadata, and release it as a resource for TKG forecasting evaluation. We formalize festival lineup forecasting as temporal link prediction between artists and festivals at future timestamps. We evaluate six TKG forecasting models on this task, analyze their capabilities and limitations, and compare them against Large Language Models applied zero-shot. Our resource complements existing TKG benchmarks by grounding evaluation in a concrete, real-world application domain.

Large language models (LLMs) are increasingly used as knowledge bases (KBs) due to the vast amount of knowledge they acquire during pre-training. While many works focus on extracting single relational triples, most real-world relations are multi-valued and require generating sets of entities.

In this paper, we investigate how LLMs represent and generate multi-valued relations. We identify the canonical order problem: The probabilistic distributions inside LLMs organize many multi-valued relations according to a canonical ordering (e.g., alphabetical or chronological). Through mechanistic analysis, we show that set generation in LLMs can be thought of in terms of three phases: (1) retrieval of candidate entities, (2) internal sorting, and (3) selection of the next element. As a result, prompts aiming to construct KBs that deviate from this internal canonical ordering lead to a markedly reduced reliability of LLMs when aiming to generate complete sets for multi-valued relations.

Relational databases are among the most widely deployed forms of structured knowledge, and natural language access to them requires grounding language onto schema entities and relations while handling the ambiguity inherent in how people phrase requests. Existing synthetic NL-to-SQL data generation methods largely ignore this ambiguity and produce oversimplified queries that fail to prepare models for the complexity of real-world structured knowledge access. We present FALCON, a framework that generates realistic, ambiguity-aware NL-to-SQL data matching the complexity of challenging real-world benchmarks, at low cost using compact open models. Our approach combines reserved-word SQL seeding and persona-based prompting to generate structurally complex queries, while alignment-based filtering preserves difficulty by distinguishing genuinely incorrect examples from complex but valid queries. Human evaluation confirms consistent high quality across model sizes, and our generated data exceeds existing benchmarks in both SQL complexity and natural language richness. Difficulty-stratified analysis shows models trained on FALCON data increasingly outperform baseline-trained models as query complexity increases, validating our pipeline’s success in generating challenging training data. When combined with a small proportion of existing benchmark data, mixed training recovers performance on simpler queries while preserving these advantages on complex ones. The model- and database-agnostic design enables organizations to generate high-complexity NL-to-SQL training data locally without external APIs.

Retrieval-augmented generation ranks passages by topical relevance or semantic similarity, but generation needs passages that actually support a target claim. We study when these targets come apart, framing the task as claim-conditioned support ranking over a constructed candidate shortlist: given a query, an explicit target claim, and a small fixed set of candidates, rank them by evidential support. We build a controlled, typed development set of 175 controlled contrastive examples spanning seven spurious-evidence cues, each pairing one supporting passage with one targeted non-support passage expressing the designated cue, plus neutral distractors. We evaluate with a corrected, tie-aware protocol that uses role-free evaluator-facing opaque identifiers and neutral tie handling. Tested query-only relevance rankers (lexical, dense, and a cross-encoder) yield tie-aware focal-pair error rates of roughly 18–34%; in the BM25 cue decomposition, several strata have high error, including negation and relation directionality; a gold-claim NLI reference—which receives information the rankers do not—reduces but does not eliminate the error. We argue that evidence-ranking evaluation should explicitly condition on the claim whose support is assessed, and sketch a typed, claim-conditioned evaluation direction that separates relevance, support, and source attributes. This is a controlled diagnostic on development data: it does not estimate real-world prevalence or provide a deployable method.

Question answering in natural language over tabular and relational data has become a key capability in the democratization of insights from structured data sources. These systems first require table retrieval to identify data that is relevant to a given natural language query, for which several methods have been introduced. In this work we present and study a fine-grained decomposition-based and global connectivity-aware table retrieval mechanism (DCTR) that handles the challenges induced by open-domain question answering over relational databases in complex usage contexts. We evaluate the effectiveness of these two mechanisms through the lens of retrieval complexity which we measure along the axes of query- and data complexity. Our complexity analyses over industry-aligned benchmarks illustrates the robustness of DCTR for highly composite queries and densely connected relational databases.

Graph-based retrieval over book-length narrative text faces a choice at construction time. Merging every mention of a character into one node erases the temporal context that separates an early portrayal from a later one; keeping mentions apart scatters a character across disconnected per-chunk nodes and leaves the consolidation to query time. We present EventSocraticKG (esokg), which (i) makes events first-class nodes in a bipartite event–entity graph, (ii) extracts them from 5W1H question–answer pairs written for each chunk instead of from the chunk text, and (iii) merges mentions across chunks by their full description, with embeddings proposing candidates and an LLM confirming them. At query time esokg serves graph structure only, with no source chunks and no LLM call. On 332 long-form questions over five books (four pre-2000 novels and one post-cutoff novel), scored by an LLM judge and re-scored by two independent judges, esokg lies within half a point of an unmerged entity–event comparator and of a full-budget entity-centric method that spends roughly twice the per-query tokens, under all three judges. The post-cutoff book separates retrieval from recall: an LLM without retrieval collapses there while every retrieval method holds, but esokg trails both graph comparators, so part of its pooled standing reflects books the models already know. Its category profile, strongest on the consistency categories and weakest on Narrative and Thematic, is largely unchanged on that book. The price is a one-time build two to three times the comparators’. What it buys is a chunk-free query path in the low per-query cost regime and a merged graph on which character-chain, shared-event, and causal-chain queries are short graph walks that need no language model.

The choice of knowledge representation (KR) determines what a knowledge base (KB) can store and how that knowledge can be retrieved and reasoned over. Recent knowledge-graph (KG) retrieval-augmented generation (RAG) systems show that *explicit symbolic structure* can complement the implicit knowledge encoded in large language models (LLMs). However, KGs capture only part of a text’s linguistic meaning and often leave **conceptual** and **temporal** context implicit, leading to recurrent reasoning failures. Existing approaches address these gaps through task-specific extensions but do not provide a unified framework for coordinating conceptual, instance-level, and temporal structure. We envision the **Knowledge State Protocol** (KSP), which coordinates canonical KRs to make known dimensions of linguistic meaning explicit. By standardizing interoperability among KRs, KSP provides a common, extensible, and open foundation for *structured* agent memory and neuro-symbolic training across modalities and languages.

Lifelong knowledge-editing benchmarks turn changes in a structured knowledge base into natural-language edits and evaluation questions. Their value depends on that conversion preserving what each fact says and which entity it concerns. We audit WikiBigEdit, a benchmark of more than 500,000 Wikidata updates, and trace its released construction pipeline to a string scan over a text rendering of Wikidata rather than a parse of its data model. From a uniform random sample of 1,500 rows, we audit the 1,458 resolvable rows against their source revisions and find three consequences: qualifier and reference values are promoted to main facts; 23.1% of resolvable rows attach correct answer text to the wrong entity identifier; and locality probes are chosen by surface-string similarity rather than semantic relatedness. Overall, 22.8% of resolvable rows carry unusable text supervision. In a matched reconstruction that holds the relation and question template fixed, similarity-selected probes show 7–12 percentage points more edit-target leakage, that is, the edited answer surfaces in a neighboring question. This effect persists under parameter editing with ROME, while the aggregate locality score remains unchanged. WikiBigEdit’s official metric does not fully register the leakage. We validate the audit with an independent parser and release the audit code and a provenance-bearing repair set. Our results do not show that published scores are incorrect; they identify structural errors those scores were not designed to detect.

Reliable evaluation of factual recall in large language models (LLMs) requires ground truth that is broad, clean, and complete. Yet existing benchmarks often rely on noisy extraction, narrow domains, or incomplete knowledge bases that penalize models for generating correct facts absent from the reference. We introduce **MUSART**, a benchmark of 112 596 factual triples over 31 domains, automatically constructed by pairing Wikipedia Good Article categories with Wikidata and selecting domain-salient relations via Personalized PageRank. To overcome knowledge-base incompleteness, we contribute an automated, character-grounded augmentation pipeline that mines verbatim facts from source articles. Evaluating twenty-one open-weight LLMs against both reference regimes reveals that (i) completeness correction systematically raises precision and lowers recall across all models, especially on multi-valued relations; and (ii) factual recall varies widely across subject domains, demonstrating why single-domain evaluation confounds model capability with subject matter. All code, data, and evaluation scripts are publicly released at https://github.com/Rajjaa/MUSART.

Customers rely on specific product attributes to compare products and make purchasing decisions, but E-Commerce catalogs are messy and unstructured, making it difficult to identify which attributes matter most and extract them at scale. Standard Attribute Value Extraction (AVE) systems treat all attributes equally, producing large, inconsistent attribute sets that do not reflect the factors consumers use to differentiate products. We introduce a two-stage LLM pipeline that first discovers a compact, ranked schema of purchase-discriminative attributes for each product category, then extracts their values from catalog text using a fine-tuned compact LLM (Qwen3-4B) with Hyper-Parallel Decoding (HPD). This pipeline achieves 85% extraction accuracy, on par with the foundational LLM it was trained on, while reducing inference costs by 92% over foundational LLMs, enabling production-scale use for product discovery and catalog enrichment. The resulting category-level structured representations effectively constitute automatically constructed product knowledge bases, providing consistent, comparable attributes across varied product categories that can ground downstream knowledge-intensive applications.

Language-model-assisted pipelines emit theorems, experiment reports, and literature claims faster than any reviewer pool can check them. This paper argues that the knowledge base such pipelines demand is not a store of extracted facts but a claim ledger: a knowledge base whose entries pair each claim with the artifact that verifies it (a machine-checked proof, a rerunnable receipt, or a resolved forecast) and whose gates fail closed, blocking an unverified claim from entering the record rather than scoring it after the fact. We ground the argument in three mechanisms already running in one research operation: a twenty-two-gate manuscript-admission stack, a Lean 4 library in which the claim is the artifact, and a sealed-prediction registry with forced resolution. We then state what a community-scale claim ledger would require of its schema, its adjudication protocol, and its incentives. The evidence is a single operation and does not establish generalization; the contribution is the knowledge-base-construction framing and the fail-closed gate architecture.

Temporal reasoning over evolving semi-structured tables poses a challenge to current QA systems. We propose an approach that recasts the task as automated knowledge base construction: (1) prompting an LLM to synthesize a 3NF-compliant relational schema from Wikipedia infobox timelines, (2) populating the schema to obtain a queryable database, and (3) generating and executing SQL queries against it, with QA accuracy serving as an extrinsic evaluation of the constructed knowledge base. In a controlled grid of three schema generators crossed with six query models, the schema source accounts for 79.5% of the exact match (EM) variance against 1.6% for the query model: replacing the schema, and the prompt scaffolding derived from it, shifts EM by 14.7 to 20.0 points, whereas replacing the query model under a fixed schema shifts it by 4.4 to 12.1. From this evidence, we distill three candidate schema-design principles: balanced normalization, semantic naming, and consistent temporal anchoring, framed as correlational hypotheses. Our best configuration (Gemini 2.5 Flash schemas + Gemini-2.0-Flash queries) reaches 80.39 EM, 11.5 points above the strongest reported baseline (68.89 EM); an open-weights configuration reaches 79.52.

Large language models deployed in high-stakes settings frequently generate plausible but ungrounded claims. Standard retrieval-augmented generation (RAG) pipelines offer limited remedy, since they retrieve isolated passages without tracking cross-document evidence relationships or quantifying uncertainty. We introduce GRACE (Graph-grounded Reflective Agent Copilot Engine), a framework that deconstructs LLM responses into atomic claims and grounds them against trusted knowledge priors within a weighted bipartite graph. Edge weights encode the closeness of each claim to the priors, enabling weighted centrality analysis that classifies claims as Grounded, Refuted, or Boundary. Such classification identifies not just hallucinations but also novel or contested claims at the frontier of the model’s knowledge. To efficiently allocate human or agent resources, we formulate a Return on Attention (RoA) objective that defers a claim to expert review only when its priority-weighted uncertainty exceeds the cost of verification. Claims verified by experts are promoted to new evidence anchors, closing a validator-LLM evolutionary loop that expands the knowledge base across iterations. We evaluate GRACE across multiple language models and on datasets spanning both general and domain-specific knowledge. Our results show that our knowledge base serves as a reliable foundation for retrieval that outperforms RAG baselines, and that the RoA framework efficiently selects valuable boundary knowledge for expert verification. These findings demonstrate that graph-structured representations combined with expert-in-the-loop verification can mitigate hallucination at the system level rather than in the generation level.

While Large Language Model (LLM)-based Natural Language Inference (NLI) systems achieve high accuracy, their decision-making processes lack auditable structures. This paper explores whether NLI can be performed using only interpretable, graph-based representations of evidence. We introduce a fully graph-based pipeline where the classifier never directly processes the input text. Instead, sentences are decomposed into atomic propositions, converted into ConceptNet triples via constrained decoding, and represented as three graphs per pair: premise, hypothesis, and a retrieved ConceptNet subgraph. These graphs are then fed into a fine-tuned 0.8-billion-parameter language model. On the SNLI dataset, our pipeline achieves 89.7% accuracy, just 1.9 points below an identically trained text-based model. On ANLI, it matches the published performance of RoBERTa-large on rounds R2 and R3 (48.0% vs.\ 48.9% and 44.9% vs.\ 44.4%) but trails by 16 points on R1, resulting in an overall gap of 9 to 14 points compared to its text counterpart. We term this gap the price of interpretability and demonstrate that it stems from representational limitations rather than data constraints. Ablation studies further reveal that graphs and text are complementary: combining both modalities achieves 92.1% accuracy on SNLI.

Understanding how scientific findings are discussed and evaluated over time provides valuable evidence for assessing their reliability. We propose the Rhetorical Citation Graph (RCG), a finding-centric graph model that represents citation events by explicitly linking scientific findings, citing articles, rhetorical functions, and citation contexts. We instantiate this model as the Plant Literature Reliability Network (PLIN), a graph of 262,807 citation events automatically constructed from the plant-health literature using Natural Language Processing methods for citation and finding extraction, and rhetorical stance classification. By integrating provenance data with the rhetorical interpretation expressed in citation contexts, the RCG provides structured evidence for contextual analyses of scientific findings and lays the foundation for longitudinal studies of their evolution, scientific consensus, and reliability. Finally, we illustrate the utility of PLIN through a case study on the curation and enrichment of the EPPO Global Database, a reference knowledge base for plant health.

We present MAGIC, a six-dimensional framework for extracting structured knowledge bases of hero and magic entity interactions from narrative texts of folktales and fairytales: each interaction between a story’s protagonist and a magical helper, villain, or object is decomposed along trigger modality, actor profile and agency balance, interaction mechanics, outcome valence, narrative function, and narrative position. We build an LLM-based extraction pipeline and apply it to a corpus of 1306 folktales collected from around the world, then show the schema supports rich structural analysis: unsupervised clustering over the extracted features recovers five recurring, readily nameable interaction archetypes – invoked and unsolicited helpers, a morally tested gatekeeper, a hero-controlled magic object, and an adversarial obstacle – that are stable across the three full-corpus extraction models (0.95 mean cluster-centroid similarity) and cut across both tale type and regional tradition rather than reproducing existing typological catalogues. We further characterize the schema’s reliability under LLM extraction (inter-model, not against a human gold standard): parse success and schema-conformance rates, a decomposition of extraction drift into field-confusion versus vocabulary-incompleteness failure modes, and cross-model field-level agreement, which varies substantially across the six dimensions and is sensitive to how many tales are compared across models. Despite this field-level variability, the archetype-level structure remains stable, suggesting that knowledge bases built from LLM extraction should be evaluated at the level of the downstream structure they support.

LLM-based systems increasingly search across sources and synthesize numerical findings, but can treat values from different sources as directly comparable. The same phenomenon may be measured by defining the quantity differently or by estimating it through different procedures. Together, these choices form a value’s operationalization. Values produced under different operationalizations can differ without creating a factual conflict. Ignoring such differences can lead systems to make unsupported comparisons or combinations, or to present one result as the single correct answer even though the appropriate choice depends on the user’s purpose. We call this operationalization collapse. This vision paper argues that operationalization is part of a numerical claim, not optional metadata. Operationalization information can show what a value means and which comparisons or combinations it supports, but it may not determine which valid result should be used to answer the user’s question. We outline a research program for recovering operationalizations, attaching them to claims, determining which comparisons and combinations those claims support, and generating answers that preserve unresolved choices.

Long-term assistants must answer questions over conversations beyond the context window while grounding responses in verifiable dialogue evidence. Existing memory systems retrieve compact representations, but these may omit critical details, while similarity search may favor related records over answer-bearing source turns. We introduce GRACE-Mem, a long-term memory architecture that separates candidate discovery, candidate reranking, and final evidence construction. It uses graph records for discovery, compressed records for reranking, and provenance-linked raw dialogue spans as final evidence. After hybrid graph retrieval, multi-hop expansion, and two-stage reranking, the top-K records seed an Evidence Curation Agent. The agent searches the raw corpus, removes irrelevant candidates, recovers missing evidence, and returns provenance identifiers to reconstruct the final context from raw dialogue. GRACE-Mem achieves 85.3% on LoCoMo and 73.2% on LongMemEval, surpassing the previously reported state of the art, AMA, by 7.9 and 3.4 percentage points, respectively.

Large Language Models (LLMs) are highly capable of natural language reasoning and appear to store a great deal of lexical knowledge, but it is still unclear how much of this knowledge they actually use when reasoning, and whether they use it in the right way. On the other hand, logic-based Natural Language Inference (NLI) systems provide transparent and formally grounded reasoning, but they need to be supplied with rich lexical knowledge to prove inferences beyond purely logical ones. In this paper, we evaluate whether LLMs can identify all lexical knowledge needed to solve NLI problems and how much this knowledge contributes to proof search in a logic-based NLI system. Our research focuses exclusively on structured lexical entailments (e.g., chinchilla ⊑ small animal) as a proxy for structured explanations for NLI problems with an entailment label. First, we curate a dataset for a new task of explaining sentential entailments with a set of lexical entailments. The dataset is used to intrinsically evaluate LLMs on generating structured lexical explanations. Then, we use NLI as an extrinsic evaluation in a simple neuro-symbolic setting, assessing whether LLMs can supply sufficient lexical relations to LangPro, a natural-logic theorem prover for natural language. The results show that the proposed task remains challenging even for hosted proprietary LLMs, and that their contribution to theorem proving is moderate: generated relations are often only partially sound and may be tailored to the specific NLI problem rather than representing generally valid lexical knowledge

Knowledge Graphs (KGs) are a practical representation of information that combines structural flexibility with expressiveness. With the recent advent of Large Language Models (LLMs), it has become possible to extract KGs from plain text. However, extracting facts from text is only one part of building a usable knowledge graph. A construction system must also reconcile entity mentions, control its relation vocabulary, integrate information across passages, and preserve a traceable connection between graph facts and their source. We present LMKG, a modular LLM-based pipeline that supports both open and ontology-constrained extraction. We also propose two construction strategies for long documents: fold, which accumulates a graph over successive chunks, and enrich, which combines whole-document and chunk-level extraction before graph refinement. We evaluate LMKG on three standard benchmarks: Text2KGBench, MINE-1, and HotpotQA. We find that our ontology conditioning substantially improves conformance to a given ontology in Text2KGBench. Our system consistently recovers facts, reaching the highest reported fact recovery of 90.80% in MINE-1. We observe degraded downstream question-answering performance on HotpotQA when supplying graphs rather than full passages.

Relation classification (RC) transforms unstructured text into structured knowledge by identifying semantic relationships between entities. As natural language data volumes rapidly increase, manual knowledge extraction has become impractical. Existing zero-shot LLM-based approaches face limitations due to sparse RC task representation in instruction-tuning datasets and inherent biases toward certain relation types. To address these challenges, we propose Binary Relevance Elicitation (BRE), a true/false prompting framework that systematically queries an LLM for each candidate relation between two entities from a closed set of relation types. BRE incorporates relevance scoring to identify the most likely relation among those classified as "true," reducing false positive predictions. We evaluate BRE using multiple open-source LLMs and achieve state-of-the-art zero-shot RC accuracy across the TACRED family of datasets. Our results demonstrate performance improvements with newer model releases that are independent of parameter count, suggesting information representation advances within LLMs drive zero-shot RC capabilities.

Despite their generalization capabilities, large language models (LLMs) have counterintuitively underperformed smaller pre-trained language models (PLMs) on few-shot relation extraction (RE), the task of identifying semantic relationships between entities in text. This gap is primarily due to a systematic over-prediction bias in LLMs. We propose Distill-RE, a two-stage knowledge distillation framework in which a teacher LLM generates chain-of-thought reasoning over existing training examples to supervise two student models: a classification model that predicts relations and a confirmation model that filters false positives. Across student scales from 3B to 70B, Distill-RE establishes a new state-of-the-art among RE models, with its top-performing configuration outperforming the strongest baseline by over 31.5 micro-F1 points averaged across TACRED, TACREV, and Re-TACRED at 8-, 16-, and 32-shot settings. Notably, without synthesizing new relation instances, it outperforms most data augmentation baselines and comes within 1.8 micro-F1 points on average of the strongest.

While large language models (LLMs) have been introduced to knowledge graph completion (KGC) to improve multi-hop reasoning, most existing approaches still rely on embedding models for candidate pruning, which limits their structured reasoning capability. In contrast, fully path-based reasoning better reflects the nature of graph inference but suffers from extremely low sample efficiency. We propose AdaG-Reason, a path-based LLM-KGC framework. We formulate reasoning data synthesis as a non-stationary multi-armed bandit problem and introduce Active Reasoning Distillation to select informative trajectories by jointly considering distribution alignment, relation difficulty, and uncertainty. We further extend this bandit-based sampling principle to the reinforcement learning phase, adaptively allocating rollout queries by per-relation difficulty. For training, we propose Implicit Reasoning Distillation optimized via Step-wise Group Relative Policy Optimization (GRPO), enabling direct and stable learning of pure graph navigation policies without value function estimation. Experiments on YAGO3-10 and NELL-995 show that AdaG-Reason achieves competitive MRR and state-of-the-art Hits@1 on YAGO3-10, demonstrating the effectiveness of active data selection and pure policy learning for path-based LLM-KGC.

Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or rely on aggregate ranking metrics (Recall, MRR, nDCG) that assume item-level independence, providing little guidance on redundancy or complementarity between retrievers. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We introduce principled metrics grounded in information theory to quantify retrieval quality, redundancy/synergy, and marginal contribution. Although the framework is designed to be applicable to any setting where a ground-truth answer can construct a pseudo chunk distribution, the present empirical study focuses on multi-hop question answering - a demanding setting where retriever complementarity is especially pronounced. On four corpora and thirteen SOTA retrievers spanning lexical, dense, and graph-based families, we show that (i) the MI-based divergence metric captures retriever quality dimensions not reflected by Recall or MRR, (ii) graph-based retrievers are redundant with each other while BM25 remains informationally complementary, and (iii) in the multi-hop QA setting, a carefully chosen ensemble guided by MI-based attribution can outperform any single retriever. We position these findings as task-specific insights and provide a reusable framework for retriever analysis that practitioners can re-estimate on their own target domains.

Pretrained language models (LMs) have significantly advanced a variety of knowledge engineering tasks and have shown promise as sources of factual knowledge. While prior work has studied factual knowledge through probing and prompting, reliably constructing complete knowledge bases from the parametric knowledge of LMs remains challenging. The AKBC 2026 Shared Task invited participants to construct knowledge bases directly from LMs, given specific subjects and relations. Unlike existing probing benchmarks, the challenge imposed no simplifying assumptions on relation cardinality, allowing a subject entity to be associated with zero, one, or multiple objects, including long answer sets. To ensure accessibility, the challenge featured a single track using open-weight LMs with a total neural parameter budget of up to 32B. Nineteen submissions were received, of which twelve were ranked, exploring a variety of approaches including relation-specific prompting, evidence aggregation, and model selection.

Large language models (LMs) store vast factual knowledge, yet turning it into a reliable, set-valued knowledge base, where a subject may have zero, one, or many objects, remains hard. The AKBC 2026 shared task (5th LM-KBC edition) poses this in a strict closed-book setting: an open-weight model of at most 32B parameters, with no retrieval, no external knowledge, and no fine-tuning. We start from the metric and observe that it rewards abstention: since an empty prediction scores a perfect F1 against an empty gold set, predicting nothing everywhere already reaches 0.169 relation-averaged macro-F1, and on the null-heavy relations even beats the official baseline. We therefore treat when to answer as a first-class decision and build a per-relation pipeline that grounds each prompt in the relation’s exact scope, decides abstention from the sampled set rather than by instruction, and takes a median over samples for numeric relations. Our Gemma-4-31B system (30.7B, the strongest in-budget model we could access) scores 0.461/0.488 (relation-averaged/micro) on validation from a single sample, beating the official baseline on five of the six relations, while a smaller 4-bit Qwen2.5-14B, even with self-consistency voting, reaches only 0.415: in this comparison a larger single-sample model outscores a smaller voted one. Generation is cached, so decoding reproduces without a GPU. We release all code, prompts, and cached generations: https://github.com/minhaz-42/lm-kbc-2026

The AKBC 2026 shared task (Kalo et al., 2026) asks a system to complete the object set of a subject and a relation out of a language model’s parameters, with no retrieval, no training, and no more than 32B parameters. Our system drives one 29.6B backbone through three stages. Several elicitation channels per relation build a candidate pool, exhaustive one-token Yes/No probes verify whatever part of the answer space can be enumerated, and a per-relation decision layer emits the object set. The backbone writes a reasoning channel before its user-facing answer, so a probe prompt has to append the answer-channel header before its first generated token is a verdict. Forcing that channel also makes auxiliary generations cheap, which is what puts roughly 26k border probes inside the budget. Most decision constants are fitted on validation gold under a rule that displaces an incumbent only on a strict improvement of more than one row, and the fitting procedure itself is cross-validated, which cut one city gain from +0.070 to +0.020 before it was adopted. The system scores 0.6553 official macro-F1 against 0.6415 for a prompting baseline on the same backbone, and the gap is uneven: the machinery is worth +0.1662 on awards and costs 0.0204 on hasCapacity, where plain prompting wins. We also report the ceiling we could not move. Twenty-one method families have been fitted against hasCapacity and none of them beats a mass vote over the pool. Harvesting the model’s own reasoning traces raises pool coverage from 0.582 to 0.724 and converts none of it.

We present the REAP system for the AKBC Shared Task 2026 on constructing knowledge bases from language models in a closed-book setting, subject to a budget of at most 32B parameters and no model fine-tuning. Our system combines structured chain-of-thought reasoning, relation-specific query strategies, and a reasoning-based empty-set gate to elicit parametric knowledge, followed by direct extraction into valid JSON arrays. On the test set, the system, built on the Mistral-Small-24B-Instruct-2501 model, achieves a macro-F1 score of 0.62, with particularly strong results on countryLandBordersCountry (F1 = 0.95), companyTradesAtStockExchange (F1 = 0.73), and hasArea (F1 = 0.77). Our code is publicly available at https://github.com/yammdd/AKBC-Shared-Task-2026.

The AKBC 2026 shared task asks a language model to recover complete object sets closed-book. SetProbe anchors each subject, samples several candidate sets, and aggregates them per relation, reaching 0.513 macro F1. Sample count, aggregation rule, and abstention thresholds each move the score by less than the evaluation set’s own 0.045 confidence interval; only the choice of model does. The binding constraint is elementary: closed-book prediction cannot recover an object that appears in no sample.

This paper describes the first place system of the AKBC 2026 shared task on predicting complete knowledge base entries from language models, built on a single 31.3B parameter base model in completion mode with no training and no retrieval. It scores 0.7060 on the official test set; the runner up scores 0.6908, and a composite taking the best per relation score of every other ranked team would reach 0.7011.

The largest single improvement I found is not an algorithm. Asking the model to recall facts about an entity before committing to an answer, rather than asking for the answer directly, moved hasArea from 0.8100 to 0.8700. Three earlier configurations, across two frames and two aggregation methods, had all returned exactly 0.8100, which I had read as evidence that the answers were absent from the candidate pool. That reading was incorrect.

The main analytical result is a correction to how this field measures headroom, and it corrects my own earlier conclusion as much as anyone else’s. Oracle coverage analyses report that hasCapacity is selection bound: the recitation frame puts an acceptable answer in the pool for 84.5 percent of validation subjects while the aggregator realises 36.1 percent. That inference does not survive a null control. A pool of one hundred guesses spanning one decade contains about twenty one mutually exclusive 5 percent tolerance windows, so it contains something acceptably close to almost any plausible target whether or not the model knows the answer. Permuting gold values across subjects measures that coincidence rate directly: 0.355 for hasCapacity against 0.031 for hasArea. Chance corrected, the findable headroom beyond the rank one answer falls from 0.443 to 0.203. Three instruments that are not functions of vote frequency then agree that the relation is knowledge bound.

We describe our system for the AKBC Shared Task 2026, which asks participants to build knowledge base entries from open-weight language models under a strict 32B total-parameter budget, closed-book setting, and no fine-tuning. Our system combines three complementary families of techniques: (i) diversity-based generation, which uses multi-question paraphrase prompting and temperature sampling with self-consistency voting to maximize recall; (ii) gold-caliber post-processing, a set of generic rules that align model outputs to the gold annotation caliber (e.g., collapsing regional stock exchanges, filtering contested territories, and calibrating capacity estimates by rounding patterns); and (iii) confidence-based gating, which uses log-probability signals to reject low-confidence answers and to verify candidates through independent knowledge pathways. On the validation set our best single-model configuration (Mistral-24B, 24B parameters) reaches a macro-F1 of 0.6562. We additionally analyze which strategies transfer across base models and which fail, providing a practical recipe for closed-book KB construction within tight parameter budgets.[Code is available at <https://github.com/LexieeWei/akbc2026>.]

Extracting facts from the parametric memory of Large Language Models (LLMs) for Automated Knowledge Base Completion (AKBC) under closed-book constraints remains a significant challenge. We describe our system for the 2026 AKBC Shared Task, where retrieval, external knowledge sources and fine-tuning are not permitted. We first evaluate 18 open-weight models on the training set to check how much the choice of the model matters for this task. The results differ significantly, from 0.186 to 0.499 macro F1, and the best model is not the largest one. Moreover, the models that score well on coding benchmarks tend to score worse on knowledge base construction. We then introduce Pre-training Aware Directed Reasoning (PADR), a set of relation-specific reasoning pipelines optimized for the best evaluated model, Mistral Small 3.2 24B. PADR improves the macro F1 score on the training set from 0.499 to 0.614. On the shared task’s test set, PADR reaches 0.629 macro F1, more than double the official baseline’s 0.296. The code is available at https://github.com/MikolajTym/akbc-padr.

We describe our AKBC 2026 closed-book knowledge base construction system: Gemma-3-27B-pt, raw text completion, 20 candidates per query, and relation-specific aggregation without an arbiter. It uses 27.4B parameters and scores 0.6670 row-averaged Macro F1 on the official test, placing sixth. Experiments distinguish this final pipeline from an earlier Mistral system with an 8B arbiter. Backbone and access format outweighed prompt-level interventions: under matched prompt formats, the pretrained checkpoint exceeds its instruction-tuned sibling by 0.11 on validation. Numeric example configurations primarily shift retrieval; procedural wording with five added JSON examples changes the selected answers by only one row. Verification instructions can shift rejection. A biography-first city prompt format affects both, raising validation F1 from 0.55 to 0.61 at a fixed vote gate. Gating, information outside the model, and seed stability help interpret transfer. These findings have counterexamples; leaderboard comparisons describe development outcomes, not an untouched final evaluation.

Large Language Models (LLMs) encode substantial factual knowledge, but reliably extracting complete knowledge-base entries remains challenging when relations differ in cardinality, semantics, and missingness. We present COVER-KBC, our system for the AKBC Shared Task 2026, which formulates closed-book knowledge extraction as relation-typed, evidence-centric inference. Each relation is compiled into an inference program that controls elicitation, verification, stopping, and final selection. Model observations are accumulated in a provenance-aware evidence state and scored using independent support, calibrated verification, contradiction, and disagreement. To distinguish candidate correctness from answer-space completeness, COVER-KBC estimates residual search need and uses a budget-aware controller to adaptively acquire further evidence when necessary. Relation-specific finalization handles heterogeneous output structures and difficult numeric or nullable cases. Using a single Mistral-Small-3.2-24B checkpoint without additional training or external retrieval, COVER-KBC achieves a Macro-F1 of 0.5878 on the hidden test set. Our implementation is available at https://github.com/vquclinh/cover-kbc.

Pretrained language models hold factual knowledge in their parameters, and the LM-KBC challenge measures how much of it can be turned into a knowledge base: given a subject and a relation, a system must return every object that completes the pair, and nothing at all when there is none. The 2026 edition forbids any lookup and caps all models used together at 32B parameters. Because the model may be chosen freely for the first time, a system has three things to decide: which model to run, how to put the question to it, and how to choose an answer from what comes back. We propose MAVE, which puts each item to one open-weight 24B model as several different questions and gives each question one vote. Each of the six relations applies its own rule for how much agreement it takes before an object is emitted. It reaches an official test macro-F1 of 0.6908, which places it second on the final leaderboard.

Knowledge base construction requires complete answer sets, not just one plausible fact. We present MuseList, a closed-book system for the six-relation AKBC Shared Task 2026. A single open-weight 29.8B-parameter Muse Glimmer model generates candidate answers in several formats, such as recipient lists, recipient–year tables, and venue records. All such records are model-generated; the system retrieves no external documents. Relation-specific rules combine these candidates, select numerical answers using token-level confidence signals, and check proposed reciprocal borders with three separately prompted roles. The complete system scores 0.6700 hidden-test macro-F1 and ranks fourth among twelve systems. Its development validation score is 0.69160; this split informed method selection and is not an untouched evaluation. We describe the method and release an archive that reconstructs the scored submission exactly. Fresh-call support is separate and partial: it regenerates later-stage responses, not an end-to-end submission.

We present our solution to the AKBC LM-KBC 2026 Shared Task, where we achieved a score of 0.6836 on the official leaderboard, more than double the baseline of 0.2964. This task requires generating all possible objects for subject-relation pairs using a language model with up to 32 billion parameters, with no retrieval or fine-tuning involved. We show that the prompting methodology is as important as the model architecture: while the instruction-tuned model version achieves 0.5681 with its chat interface, the base model achieves 0.6836 when prompted using text completion. Thus, instruction tuning decreases factual recall; we refer to this effect as the instruction tax, which is caused by the model’s weights rather than the prompting method. Our final solution is fairly straightforward: we prompt the base model with some examples from the training data, generate 64-256 outputs, and retain only the repeated ones. Two specialized methods are then applied for each relation: one improves the recall of large answer sets, and the other verifies a person’s vitality before predicting their place of death.

This paper presents a system developed for the fifth AKBC Shared Task on Knowledge Base Construction from Language Models (LM-KBC 2026). The task involves predicting sets of object-entity for (subject, relation) pairs across six relation types, ranging from single-valued relations to relations with potentially hundreds of recipients. All predictions are generated in a closed-book setting (CITATION), utilizing up to 32 billion parameters across all neural components, without resorting to fine-tuning or retrieval augmented techniques. The development process included two stages: validating the technical methodology using the officially specified baseline model (Qwen3.5-9B), followed by comparing nine open-weight models using the same evaluation framework. This process led to the selection of Gemma 3 27B as the final system. The paper presents the main output-parsing errors identified and resolved, together with an evaluation of self-consistency and majority-voting aggregation for each relation type. The final system improved macro-F1 score from 0.313 to 0.531 (a +70% relative improvement) through workflow optimization and model selection.

We report a system for the LM-KBC shared task under a strict single-repository parameter budget of 32B, which rules out both model ensembling and scaling to a larger model. Starting from a rationalized chain-of-thought (CoT) pipeline that reached macro-F1 0.522 with Gemma-4-31B-it, we show that swapping to a smaller, differently-trained model (Qwen3.5-27B) under an identical pipeline recovers 24 percentage points of exact-match accuracy on a purely numeric relation (surface area), while costing accuracy on relations requiring broad factual recall. This shows that parametric knowledge coverage, not reasoning capability, is the limiting factor at this model scale. We then present four negative or reversed results (self-consistency sampling on numeric relations, a confabulation-guard prompt, and two cases of non-comparable validation splits) as diagnostic evidence for why scaffolding choices that help one relation type hurt another. We introduce CoRe routing (Construct versus Recall), which assigns each relation to CoT or direct answering according to whether its answer must be constructed or merely recalled. CoRe reaches macro-F1 0.548 on validation and 0.541 on the test split.

Large language models represent a promising alternative for automating structured knowledge bases construction, but their reliance on parametric knowledge makes them susceptible to hallucinations and incomplete information. The Automated Knowledge Base Construction competition investigates the use of large language models to construct structured knowledge bases from their parametric knowledge alone, without fine-tuning or external retrieval. Given a subject–relation pair, participants must predict the complete set of correct objects across several relation types. Our approach proposes an ensemble of four prompting strategies: question-style, masked-style, few-shot chain-of-thought, and least-to-most prompting. It aggregates their predictions using relation-aware majority voting. The resulting system achieves a Macro-F1 of 0.6651 on the test set, substantially outperforming the provided baseline of 0.294.

We describe a system for the AKBC Shared Task 2026. The task is to predict complete sets of knowledge base objects from parametric memory alone, with no retrieval and no fine-tuning, across six relations and within a 32B parameter budget. Our system combines relation-specific multiple prompting, consistency-based aggregation, and error-driven prompt refinement. For each relation, multiple prompt variants and samples elicit complementary predictions, with cross-prompt agreement reducing wording-specific errors and cross-sample agreement filtering unstable guesses. Aggregation is tailored to each relation, including cluster-median aggregation for numeric values and voting for categorical and set-valued outputs. The system achieves a validation macro-F1 of 0.5577 and a test macro-F1 of 0.5699, compared to organizer baselines of 0.3131 (validation) and 0.2964 (test).

We describe a system for the LM-KBC 2026 shared task on closed-book knowledge base construction. Given a subject and relation, the system tries to predict all corresponding objects without retrieval or additional training and remains within the 32-billion-parameter limit. The system uses the base model gemma-4-31B with separate prompting and aggregation rules for six relations; few-shot demonstrations and selection thresholds come from the train set and are checked on the validation set. It obtains 0.6975 global F1 on the official validation set, compared with 0.308 for the organiser baseline on validation. The largest per-relation gains over our earlier baseline configuration come from a company-listing prompt with explicit abstention and a log-stratified area prompt, plus a vote-share abstention threshold for empty-heavy relations (like personHasCityOfDeath) that recovers most of the empty-case precision. Across more than 30 prompting, verification, and aggregation configurations, further inference rarely recovers facts that are absent from the generated candidates. These results indicate that relation-specific elicitation improves closed-book extraction, while candidate coverage and selection remain the main limitations. Code, prompts, and configuration files are available at https://github.com/ruggsea/lmkbc2026-submission.