Ahsan Bilal
Author directory2026
Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
Atta UL Asad | Ahsan Bilal | Muhammad Ali | Muhammad Haseeb | Dean F. Hougen
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Atta UL Asad | Ahsan Bilal | Muhammad Ali | Muhammad Haseeb | Dean F. Hougen
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Retrieval-augmented generation (RAG) systems can precompute and store key-value (KV) caches of retrieved documents to avoid re-encoding the same context for every query. Quantizing these offline caches reduces storage cost, but existing work largely evaluates compression through accuracy alone and does not ask whether the generated answers remain faithful to the retrieved evidence. This distinction is important because a model can preserve the correct answer string while weakening its grounding in the provided context. We audit Qwen2.5-7B-Instruct under BF16, INT8, and INT4 offline KV-cache settings on RGB and HotpotQA, measuring both answer accuracy and faithfulness using a hallucination detector, NLI entailment, and an LLM judge. INT8 is nearly lossless across accuracy and faithfulness metrics. In contrast, INT4 reduces accuracy and, more critically, degrades faithfulness even on examples where the answer remains factually correct: over 90% of faithfulness flips are negative. The degradation increases under noisier retrieval and larger retrieval depth. These results show that compressed RAG systems cannot be validated by EM or F1 alone; faithfulness must be audited before low-bit offline KV caches are deployed.
Logic-Level Evaluation of Logical Table-to-Text Generation
Lena Trigg | Dean F. Hougen | Ahsan Bilal
Proceedings of the 30th Conference on Computational Natural Language Learning
Lena Trigg | Dean F. Hougen | Ahsan Bilal
Proceedings of the 30th Conference on Computational Natural Language Learning
Logical Table-to-Text (LT2T) generation aims to produce natural-language sentences that are logically faithful to structured tabular data. While recent Large Language Models (LLMs) show high performance on aggregate fidelity metrics, these scores provide only a coarse view of performance, obscuring specific logic-type reasoning failures and models’ meta-logical awareness. We propose an operation-aware diagnostic framework that evaluates four core competencies: (1) Logical Form (LF) execution accuracy, (2) fidelity of LF-conditioned generation, (3) logic-type identification, and (4) LF-free generation.We apply this framework to a suite of frontier LLMs and perform fine-grained analysis across logic types such as aggregation, ordinal, and superlative reasoning. Our results show that LT2T fidelity assessment can be unstable; the choice of verifier and logic type can substantially alter conclusions and model rankings. Crucially, we identify a meta-logical gap: models often generate faithful statements while failing to identify the underlying operation.