Diffusion Language Models (DLMs) have emerged as a promising alternative to autoregressive language models, showing strong non-autoregressive generation abilities. Although the masked denoising objective trains DLMs to recover tokens at arbitrary positions, current long-context usage still mostly follows a prefix-based paradigm: the full context is given as a prefix, and the answer is generated only at the end, leaving the model’s infilling ability underutilized. To investigate whether DLMs can read long contexts by infilling, we introduce Parallel Infilling Reading (PIR), a training-free method that inserts extraction slots into long contexts to generate intermediate reading states. Our empirical study shows that naive PIR improves over prefix-based DLM inference, but suffers from slot repetition, where multiple slots generate redundant evidence. To mitigate this issue, we propose Slot-Isolated PIR, which blocks direct slot-to-slot attention, and PIR-SFT, which uses fine-grained evidence supervision to teach DLMs effective evidence extraction while reading. Experiments on long-context question answering benchmarks show that our methods improve answer accuracy and generation efficiency, with increasingly larger gains as the context length grows.

Advancements in Large Language Models (LLMs) have extended their input context length, yet they still struggle with retrieval and reasoning in long-context inputs. Existing methods propose to utilize the prompt strategy and Retrieval-Augmented Generation (RAG) to alleviate this limitation. However, they still face challenges in balancing retrieval precision and recall, impacting their efficacy in answering questions. To address this, we introduce CAFE, a two-stage coarse-to-fine method to enhance multi-document question-answering capacities. By gradually eliminating the negative impacts of background and distracting documents, CAFE makes the responses more reliant on the evidence documents. Initially, a coarse-grained filtering method leverages retrieval heads to identify and rank relevant documents. Then, a fine-grained steering method guides attention to the most relevant content. Experiments across benchmarks show that CAFE outperforms baselines, achieving an average SubEM improvement of up to 22.1% and 13.7% over SFT and RAG methods, respectively, across three different models. Our code is available at https://github.com/RUCAIBox/CAFE.