Tan-Minh Nguyen

Author directory

2026

While retrieval-augmented generation (RAG) grounds large language models (LLMs) in external knowledge, ensuring faithful evidence utilization in complex domains like law remains challenging. LLMs often ignore retrieved facts or indiscriminately copy from noisy contexts. Existing context-aware decoding methods attempt to mitigate this by amplifying the influence of retrieved content, but they treat the context as a monolithic block—amplifying irrelevant spans alongside evidence—and intervene at every decoding step, incurring substantial inference overhead. To address this, we propose SaCoD (Selective and Confidence-aware Decoding), a training-free inference framework. SaCoD leverages the LLM’s internal attention signals to dynamically isolate salient evidence paragraphs at each step, restricting a customized copy mechanism strictly to relevant facts. Furthermore, an entropy-based gate activates this intervention only when the model is uncertain, bypassing copy operations for deterministic tokens to preserve linguistic fluency and cut computational overhead. Experiments on four legal question-answering benchmarks across three open-weight LLMs demonstrate that SaCoD significantly improves faithfulness and response correctness while substantially reducing inference latency compared with prior context-aware methods.

2025