Large language models (LLMs) show strong performance in medical question answering, but their computational costs limit deployment under resource constraints. Therefore, an increasing number of studies focus on distilling knowledge from LLMs into smaller models to improve performance in medical question answering. However, existing methods often fail to identify truly useful knowledge to distill, making it difficult to focus distillation on the student’s real weaknesses. As a result, much of the distillation budget is spent on already acquired knowledge rather than on weaknesses that still limit performance. To address this problem, we propose LEAD, a Learnable Error-Aware Distillation framework that precisely improves the weaknesses of student models. First, LEAD identifies questions answered incorrectly by the student model to determine which weaknesses require teacher guidance. Then, LEAD generates teacher rationales and similar samples to construct candidate supervision for these errors. Through supervision refinement, reliable and learnable training samples are retained from the candidate supervision. Finally, LEAD trains the student on a balanced sample mixture to correct learnable errors while preserving previously acquired knowledge. Experiments on four medical question answering benchmarks show that LEAD outperforms existing distillation methods.

Retrieval-Augmented Generation (RAG) mitigates hallucinations in large language models by incorporating external knowledge. However, retrieval does not always return relevant documents and may return noisy ones. Indiscriminately retrieving and utilizing this external knowledge can interfere with the model’s originally correct reasoning. In this work, we propose Dual-Decision Retrieval-Augmented Generation (D²-RAG), which integrates multi-dimensional uncertainty estimation to decide whether to retrieve and employs adaptive contrastive decoding to handle retrieved contexts of varying quality. Specifically, we first integrate uncertainty estimation scores that assess model uncertainty from multiple perspectives, construct them into a comprehensive feature vector, and train a lightweight retrieval decision model to accurately identify the model’s knowledge boundaries and determine whether to retrieve. Subsequently, we dynamically adjust the contrastive decoding strategy based on the utility of retrieved contexts to enhance the utilization of relevant contexts while suppressing interference from noisy contexts. Extensive experiments on four medical question-answering datasets demonstrate that D²-RAG significantly outperforms baselines, enabling retrieval-augmented Llama3.1-8B to surpass non-retrieval-augmented Llama3.1-70B on the MedMCQA dataset. The source code is available on https://github.com/zakelawen/d--rag.

Large language models (LLMs) have demonstrated strong reasoning abilities, but hallucinations and unverifiable intermediate reasoning steps remain key challenges. In knowledge graph question answering (KGQA), entity-relation paths from knowledge graphs (KGs) provide structured and verifiable evidence for grounded reasoning. However, existing KG-enhanced LLM methods commonly linearize all retrieved reasoning paths into textual prompts, introducing noise and underexploiting their ordered structure. To address these limitations, we propose PathSoft, a hybrid soft prompting framework that combines textual path evidence with soft prompts. PathSoft first employs PathMLP to select reliable reasoning paths, and then uses PathSoftAdapter to convert the selected paths into continuous soft prompts while preserving their ordered path structure. Finally, the resulting soft prompt tokens are combined with the textual prompt at the embedding layer of the frozen LLM, enabling it to exploit both readable path evidence and path structure information. Experiments on WebQSP and CWQ show that PathSoft outperforms existing KGQA baselines. The code is available at https://github.com/tengyan1786-blip/PathSoft.