Large Language Models (LLMs) have demonstrated remarkable abilities, one of the most important being in-context learning (ICL). With ICL, LLMs can derive the underlying rule from a few demonstrations and provide answers that comply with the rule. Previous work hypothesized that the network creates a task vector in specific positions during ICL. The task vector can be computed by averaging across the dataset. It conveys the overall task information and can thus be considered global. Patching the global task vector allows LLMs to achieve zero-shot performance with dummy inputs comparable to few-shot learning. However, we find that such a global task vector does not exist in all tasks, especially in tasks that rely on rules that can only be inferred from multiple demonstrations, such as categorization tasks. Instead, the information provided by each demonstration is first transmitted to its answer position and forms a local task vector associated with the demonstration. In some tasks but not in categorization tasks, all demonstrations’ local task vectors converge in later layers, forming the global task vector. We further show that local task vectors encode a high-level abstraction of rules extracted from the demonstrations. Our study provides novel insights into the mechanism underlying ICL in LLMs, demonstrating how ICL may be achieved through an information aggregation mechanism.

Intermediate-layer representations in large language models (LLMs) contain task-relevant information, yet direct linear readouts often fail to recover the model’s eventual choice. This gap limits both interpretability and early-exit methods for candidate-scoring workloads. We study single-token decision settings – such as multiple-choice and boolean QA – where the answer is a single token at a position fixed by the task format, and we read out the model’s prediction there as its highest-probability token over the full vocabulary. We propose SimLens, a training-free readout that, at a chosen intermediate layer, keeps only the start token and the candidate answer token and forwards this reduced two-token state through the model’s remaining upper layers. To identify the layer at which the full sequence can be safely truncated, we distill LinearSimLens as a lightweight entropy monitor and combine the two in SimExit. Across different models, SimLens produces earlier and more faithful candidate readout than direct linear lenses under fair-compute accounting, while SimExit delivers a modest but stable speed–accuracy trade-off on the two models used end-to-end. Ablations show that the two retained tokens play complementary roles as global condition and semantic anchor.