Large language models drop in performance when their context contains semantic distractors: context topically related to the question but off the reasoning chain. Prior mechanistic work tracks a single injected distractor token, missing failures in which the model predicts other context tokens or out-of-context but topically related ones. We instead track the model’s actual top non-gold prediction, providing a localization target for every observed top-1 failure in the analyzed subset. Failures converge on a small set of mid-to-late-layer answer-selection heads (AS-Heads): patching their clean outputs into distracting runs recovers 60-86\\%$ of failed predictions across four base models from two families (random heads recover $\leq 8\\%$), with the mechanism also replicating in two instruction-tuned models. Clean-prompt ablations indicate that these heads also contribute to normal answer selection, so their role is not specific to distractors. Redirecting their attention onto an entity span causes the model to transcribe that span at $22$-$83\\%$ (vs. $\leq 7.4\\%$ for random heads), establishing a causally consequential intervention point. The localized heads transfer across MuSiQue, HotpotQA, 2WikiMultiHopQA, and non-Wikipedia RAG settings. At inference, their attention provides a training-free context-filtering signal that lifts F1 by up to $+35$ points on disrupted samples and is competitive with retrieval-head methods.

With the development of large language models (LLMs), there has been an increasing need for significant advancements in handling long contexts. To enhance long-context capabilities, constructing high-quality training data with long-range dependencies is crucial. Existing methods to select long-context data often rely on sentence-level analysis,which can be greatly optimized in both performance and efficiency. In this paper, we propose a novel token-level framework, ​LongAttn​, which leverages the self-attention mechanism of LLMs to measure the long-range dependencies for the data. By calculating token-level dependency strength and distribution uniformity of token scores, LongAttn effectively quantifies ​long-range dependencies​, enabling more accurate and efficient data selection. We filter LongABC-32K from open-source long-context datasets (ArXiv, Book, and Code). Through our comprehensive experiments, LongAttn has demonstrated its excellent ​effectiveness​, ​scalability​, and ​efficiency​. We will release our code and the high-quality long-context dataset LongABC-32K in the future.

Generating Wikipedia articles autonomously is a challenging task requiring the integration of accurate, comprehensive, and well-structured information from diverse sources. This paper introduces the Memory Organization-based Generation (MOG) framework, a novel approach to address these challenges by leveraging a hierarchical memory architecture. MOG extracts fine-grained memory units from web documents, recursively organizes them into a Wikipedia-style hierarchical structure, and uses this structure to guide the generation process. This ensures alignment between memory and the article outline, improving both informativeness and verifiability while minimizing hallucinations. Additionally, a citation module is implemented to enhance traceability by linking every generated sentence to specific memory units. Evaluations on our newly created WikiStart dataset demonstrate that MOG outperforms baseline methods in producing informative and reliable articles, making it particularly robust in real-world scenarios.