Tanvir Ahmed
Author directory2026
Closed-Book Knowledge Base Construction with Calibrated Abstention and Numeric Self-Consistency
Tanvir Ahmed
Proceedings of the 11th Workshop on Automated Knowledge Base Construction (AKBC 2026)
Tanvir Ahmed
Proceedings of the 11th Workshop on Automated Knowledge Base Construction (AKBC 2026)
Large language models (LMs) store vast factual knowledge, yet turning it into a reliable, set-valued knowledge base, where a subject may have zero, one, or many objects, remains hard. The AKBC 2026 shared task (5th LM-KBC edition) poses this in a strict closed-book setting: an open-weight model of at most 32B parameters, with no retrieval, no external knowledge, and no fine-tuning. We start from the metric and observe that it rewards abstention: since an empty prediction scores a perfect F1 against an empty gold set, predicting nothing everywhere already reaches 0.169 relation-averaged macro-F1, and on the null-heavy relations even beats the official baseline. We therefore treat when to answer as a first-class decision and build a per-relation pipeline that grounds each prompt in the relation’s exact scope, decides abstention from the sampled set rather than by instruction, and takes a median over samples for numeric relations. Our Gemma-4-31B system (30.7B, the strongest in-budget model we could access) scores 0.461/0.488 (relation-averaged/micro) on validation from a single sample, beating the official baseline on five of the six relations, while a smaller 4-bit Qwen2.5-14B, even with self-consistency voting, reaches only 0.415: in this comparison a larger single-sample model outscores a smaller voted one. Generation is cached, so decoding reproduces without a GPU. We release all code, prompts, and cached generations: https://github.com/minhaz-42/lm-kbc-2026