In-context learning (ICL) adapts LLM behavior by placing task-relevant context in the prompt. While powerful, this mechanism can be inefficient: the context must be reprocessed for every query, and when it is substantially longer than the query itself, it dominates inference cost in both latency and token overhead. This paper contends that the behavioral shift induced by an in-context signal can be compiled directly into the model weights, enabling subsequent inference without including the original context in the prompt. To this end, we propose In-Context Compilation (ICC), a gradient free method that encodes a context-induced behavioral shift as a sparse closed-form edit to attention output-projection weights. ICC is calibrated solely from the activation differences between no-context and with-context forward passes on a small set of behaviorally informative examples, requiring neither backpropagation nor trainable parameters. We evaluate ICC in the few-shot setting, where labeled contexts provide controlled behavioral targets. Across diverse LLM architectures and classification and reasoning benchmarks, the ICC-adapted LLM closely approximates the performance of the same LLM prompted with the full context, while omitting that context at inference time. In some cases, it even surpasses the with-context model. Moreover, on most benchmarks, ICC is competitive with and often outperforms gradient-based PEFT methods (LoRA, DoRA, IA3, Prompt/Prefix Tuning, LoReFT) and training-free activation-injection baselines (Function Vectors, ICV), while completing compilation in 10–18s with only ∼64–120 MB of overhead.

Retrieval-augmented generation (RAG) is usually evaluated from one observed trace: the query, the retrieved context, and the generated answer. Such evaluations can tell us whether the answer is consistent with the retrieved context, but not whether the system would still give the same answer if that context were removed, updated, or contradicted. We introduce COUNTERMEM, an evaluation protocol for RAG systems that are expected to answer from retrieved evidence: it keeps the RAG configuration fixed, changes only the retrieved context, regenerates the answer, and records whether the answer changes in the expected direction. Across RAGBench, ExpertQA, and DelucionQA, COUNTERMEM complements observed-trace faithfulness by exposing failures it is not designed to see. Configurations that look equivalent under RAGAS faithfulness behave very differently once their evidence is removed, updated, or contradicted, and among RAGChecker-approved answers more than half silently miss coherent contradictions and over a third ignore coherent updates. The failures persist across generator and judge families, the behavioral detectors agree with blind human annotators as closely as the annotators agree with each other, and row-level intervention traces localize failures to retrieval, reranking, or generation. Artifact (protocol, prompts, detectors, scripts, audit traces): https://github.com/DeltaLabTLV/CounterMem.

In this paper, we introduce Attributive Masking Learning (AML), a method designed for explaining language model predictions by learning input masks. AML trains an attribution model to identify influential tokens in the input for a given language model’s prediction. The central concept of AML is to train an auxiliary attribution model to simultaneously 1) mask as much input data as possible while ensuring that the language model’s prediction closely aligns with its prediction on the original input, and 2) ensure a significant change in the model’s prediction when applying the inverse (complement) of the same mask to the input. This dual-masking approach further enables the optimization of the explanation w.r.t. the metric of interest. We demonstrate the effectiveness of AML on both encoder-based and decoder-based language models, showcasing its superiority over a variety of state-of-the-art explanation methods on multiple benchmarks.

We present Randomized Path-Integration (RPI) - a path-integration method for explaining language models via randomization of the integration path over the attention information in the model. RPI employs integration on internal attention scores and their gradients along a randomized path, which is dynamically established between a baseline representation and the attention scores of the model. The inherent randomness in the integration path originates from modeling the baseline representation as a randomly drawn tensor from a Gaussian diffusion process. As a consequence, RPI generates diverse baselines, yielding a set of candidate attribution maps. This set facilitates the selection of the most effective attribution map based on the specific metric at hand. We present an extensive evaluation, encompassing 11 explanation methods and 5 language models, including the Llama2 and Mistral models. Our results demonstrate that RPI outperforms latest state-of-the-art methods across 4 datasets and 5 evaluation metrics.

Despite the many advances of Large Language Models (LLMs) and their unprecedented rapid evolution, their impact and integration into every facet of our daily lives is limited due to various reasons. One critical factor hindering their widespread adoption is the occurrence of hallucinations, where LLMs invent answers that sound realistic, yet drift away from factual truth. In this paper, we present a novel method for detecting hallucinations in large language models, which tackles a critical issue in the adoption of these models in various real-world scenarios. Through extensive evaluations across multiple datasets and LLMs, including Llama-2, we study the hallucination levels of various recent LLMs and demonstrate the effectiveness of our method to automatically detect them. Notably, we observe up to 87% hallucinations for Llama-2 in a specific experiment, where our method achieves a Balanced Accuracy of 81%, all without relying on external knowledge.

Language models that utilize extensive self-supervised pre-training from unlabeled text, have recently shown to significantly advance the state-of-the-art performance in a variety of language understanding tasks. However, it is yet unclear if and how these recent models can be harnessed for conducting text-based recommendations. In this work, we introduce RecoBERT, a BERT-based approach for learning catalog-specialized language models for text-based item recommendations. We suggest novel training and inference procedures for scoring similarities between pairs of items, that don’t require item similarity labels. Both the training and the inference techniques were designed to utilize the unlabeled structure of textual catalogs, and minimize the discrepancy between them. By incorporating four scores during inference, RecoBERT can infer text-based item-to-item similarities more accurately than other techniques. In addition, we introduce a new language understanding task for wine recommendations using similarities based on professional wine reviews. As an additional contribution, we publish annotated recommendations dataset crafted by human wine experts. Finally, we evaluate RecoBERT and compare it to various state-of-the-art NLP models on wine and fashion recommendations tasks.

We propose the novel Within-Between Relation model for recognizing lexical-semantic relations between words. Our model integrates relational and distributional signals, forming an effective sub-space representation for each relation. We show that the proposed model is competitive and outperforms other baselines, across various benchmarks.

This paper presents the Bayesian Hierarchical Words Representation (BHWR) learning algorithm. BHWR facilitates Variational Bayes word representation learning combined with semantic taxonomy modeling via hierarchical priors. By propagating relevant information between related words, BHWR utilizes the taxonomy to improve the quality of such representations. Evaluation of several linguistic datasets demonstrates the advantages of BHWR over suitable alternatives that facilitate Bayesian modeling with or without semantic priors. Finally, we further show that BHWR produces better representations for rare words.