Large Language Models (LLMs) have demonstrated remarkable abilities, one of the most important being in-context learning (ICL). With ICL, LLMs can derive the underlying rule from a few demonstrations and provide answers that comply with the rule. Previous work hypothesized that the network creates a task vector in specific positions during ICL. The task vector can be computed by averaging across the dataset. It conveys the overall task information and can thus be considered global. Patching the global task vector allows LLMs to achieve zero-shot performance with dummy inputs comparable to few-shot learning. However, we find that such a global task vector does not exist in all tasks, especially in tasks that rely on rules that can only be inferred from multiple demonstrations, such as categorization tasks. Instead, the information provided by each demonstration is first transmitted to its answer position and forms a local task vector associated with the demonstration. In some tasks but not in categorization tasks, all demonstrations’ local task vectors converge in later layers, forming the global task vector. We further show that local task vectors encode a high-level abstraction of rules extracted from the demonstrations. Our study provides novel insights into the mechanism underlying ICL in LLMs, demonstrating how ICL may be achieved through an information aggregation mechanism.

Synthetic data has become essential for training code generation models, yet it introduces significant noise and hallucinations that are difficult to detect with current metrics. Existing data selection methods like Instruction-Following Difficulty (IFD) typically assess how hard a model generates an answer given a query (A|Q). However, this metric is ambiguous on noisy synthetic data, where low probability can distinguish between intrinsic task complexity and model-generated hallucinations. Here, we propose QAQ, a novel data selection framework that evaluates data quality from the reverse direction: how well can the answer predict the query (Q|A)? We define Reverse Mutual Information (RMI) to quantify the information gain about the query conditioned on the answer. Our analyses reveal that both extremes of RMI signal quality issues: low RMI indicates semantic misalignment, while excessively high RMI may contain defect patterns that LLMs easily recognize. Furthermore, we introduce a selection strategy based on the disagreement between strong and weak models to identify samples that are valid yet challenging. Experiments across three datasets spanning code generation (WarriorCoder, Magpie-Qwen2.5-Coder-Pro-300K) and math reasoning (OpenR1-Math-220k) demonstrate that selecting just 25% of data using stratified RMI matches full-data performance while being consistently competitive with or better than existing data selection methods. Our approach highlights the importance of bidirectional semantic coherence in synthetic data curation, offering a scalable pathway to reduce computational costs without sacrificing model capability. Code is available at https://github.com/XXSg559/QAQ.

Intermediate-layer representations in large language models (LLMs) contain task-relevant information, yet direct linear readouts often fail to recover the model’s eventual choice. This gap limits both interpretability and early-exit methods for candidate-scoring workloads. We study single-token decision settings – such as multiple-choice and boolean QA – where the answer is a single token at a position fixed by the task format, and we read out the model’s prediction there as its highest-probability token over the full vocabulary. We propose SimLens, a training-free readout that, at a chosen intermediate layer, keeps only the start token and the candidate answer token and forwards this reduced two-token state through the model’s remaining upper layers. To identify the layer at which the full sequence can be safely truncated, we distill LinearSimLens as a lightweight entropy monitor and combine the two in SimExit. Across different models, SimLens produces earlier and more faithful candidate readout than direct linear lenses under fair-compute accounting, while SimExit delivers a modest but stable speed–accuracy trade-off on the two models used end-to-end. Ablations show that the two retained tokens play complementary roles as global condition and semantic anchor.