In-context learning (ICL) adapts LLM behavior by placing task-relevant context in the prompt. While powerful, this mechanism can be inefficient: the context must be reprocessed for every query, and when it is substantially longer than the query itself, it dominates inference cost in both latency and token overhead. This paper contends that the behavioral shift induced by an in-context signal can be compiled directly into the model weights, enabling subsequent inference without including the original context in the prompt. To this end, we propose In-Context Compilation (ICC), a gradient free method that encodes a context-induced behavioral shift as a sparse closed-form edit to attention output-projection weights. ICC is calibrated solely from the activation differences between no-context and with-context forward passes on a small set of behaviorally informative examples, requiring neither backpropagation nor trainable parameters. We evaluate ICC in the few-shot setting, where labeled contexts provide controlled behavioral targets. Across diverse LLM architectures and classification and reasoning benchmarks, the ICC-adapted LLM closely approximates the performance of the same LLM prompted with the full context, while omitting that context at inference time. In some cases, it even surpasses the with-context model. Moreover, on most benchmarks, ICC is competitive with and often outperforms gradient-based PEFT methods (LoRA, DoRA, IA3, Prompt/Prefix Tuning, LoReFT) and training-free activation-injection baselines (Function Vectors, ICV), while completing compilation in 10–18s with only ∼64–120 MB of overhead.

In this paper, we introduce Attributive Masking Learning (AML), a method designed for explaining language model predictions by learning input masks. AML trains an attribution model to identify influential tokens in the input for a given language model’s prediction. The central concept of AML is to train an auxiliary attribution model to simultaneously 1) mask as much input data as possible while ensuring that the language model’s prediction closely aligns with its prediction on the original input, and 2) ensure a significant change in the model’s prediction when applying the inverse (complement) of the same mask to the input. This dual-masking approach further enables the optimization of the explanation w.r.t. the metric of interest. We demonstrate the effectiveness of AML on both encoder-based and decoder-based language models, showcasing its superiority over a variety of state-of-the-art explanation methods on multiple benchmarks.

We present Randomized Path-Integration (RPI) - a path-integration method for explaining language models via randomization of the integration path over the attention information in the model. RPI employs integration on internal attention scores and their gradients along a randomized path, which is dynamically established between a baseline representation and the attention scores of the model. The inherent randomness in the integration path originates from modeling the baseline representation as a randomly drawn tensor from a Gaussian diffusion process. As a consequence, RPI generates diverse baselines, yielding a set of candidate attribution maps. This set facilitates the selection of the most effective attribution map based on the specific metric at hand. We present an extensive evaluation, encompassing 11 explanation methods and 5 language models, including the Llama2 and Mistral models. Our results demonstrate that RPI outperforms latest state-of-the-art methods across 4 datasets and 5 evaluation metrics.