Multilingual language models can answer factual questions across languages, but their answers are not always consistent across languages, and the internal pathways used to retrieve this knowledge remain unclear. A model may rely on language-specific shortcuts, shared multilingual representations, or another language as an internal pivot. We investigate these possibilities in Llama-3.1-8B and Gemma-2-9B with sparse autoencoders (SAEs), identifying, analyzing, and causally intervening on language-associated features during factual recall. On CUS-QA (CITATION), a parallel QA dataset spanning Czech, Slovak, Ukrainian, and English, we identify SAE features associated with particular subsets of languages. We then decompose each question’s activations into these subsets to measure reliance on input-language, multilingual, or pivot-like representations across layers. Across both models, factual recall is heterogeneous: different input languages and questions exhibit different language-associated pathways. Steering substantially changes cross-lingual answer alignment and sometimes increases correctness in Llama, while it produces only small changes in Gemma. These results show that multilingual factual recall is mediated by identifiable SAE features that can be used to diagnose and control how models access knowledge across languages.

In this work, we identify three representation-level dynamics shared across diverse large language models (LLMs) during the processing of few-shot prompts. These structural behaviors are not explicitly trained, yet they emerge reliably across model families and scales to systematically shape internal representations. Using Sparse Autoencoders (SAEs) as an analytical lens, we uncover three primary phenomena: (1) a decimal periodicity effect, where specific features exhibit a repeating modulo-10 activation pattern; (2) local feature homeostasis, a compensatory push–pull dynamic in which current representations are inversely modulated by prior exemplars; and (3) adaptive feature redistribution, wherein specific features flexibly shift their semantic profiles in response to contextual constraints. We validate these mechanisms across multiple model families and prompt formats. Together, they offer fresh insights into the internal dynamics of in-context learning and raise important caveats for future SAE-based interpretability studies.

The term “linear representation hypothesis” (LRH) has appeared across diverse subfields of artificial intelligence, neuroscience, and cognitive science. But previous works have not consistently treated the LRH as a falsifiable scientific hypothesis; we analyze these inconsistencies and examine their implications for how prior theoretical and methodological results should be interpreted. Based on this analysis, we argue that claims regarding linear representations become well-defined only through careful examination of the model, representation location, feature definition, and evaluation dataset. We therefore propose a more rigorous formalization of the LRH that makes these dependencies explicit and allows the hypothesis to be evaluated as a falsifiable scientific claim. Finally, we identify some non-trivial open problems that warrant further attention from the research community.

While large language models (LLMs) are dominating the field of natural language processing, it remains an open question how well these models can perform spatial reasoning. Contrary to recent studies suggesting that LLMs struggle with spatial reasoning tasks, we demonstrate in this paper that a novel prompting technique, termed Patient Visualization of Thought (Patient-VoT), can boost LLMs’ spatial reasoning abilities. The core idea behind Patient-VoT is to explicitly integrate bullet lists, coordinates, and visualizations into the reasoning process. By applying Patient-VoT, we achieve a significant boost in spatial reasoning performance compared to prior prompting techniques. We also show that integrating bullet lists into reasoning is effective in planning tasks, highlighting its general effectiveness across different applications.

Sparse autoencoders (SAEs) are emerging as a key analytical tool in the field of mechanistic interpretability for large language models (LLMs). While SAEs have primarily been used for interpretability, we shift focus and explore an understudied question: “Can SAEs be applied to practical tasks beyond interpretability?” Given that SAEs are trained on billions of tokens for sparse reconstruction, we believe they can serve as effective extractors, offering a wide range of useful knowledge that can benefit practical applications. Building on this motivation, we demonstrate that SAEs can be effectively applied to in-context learning (ICL). In particular, we highlight the utility of the SAE-reconstruction loss by showing that it provides a valuable signal in ICL—exhibiting a strong correlation with LLM performance and offering a powerful unsupervised approach for prompt selection. These findings underscore the versatility of SAEs and reveal their potential for real-world applications beyond interpretability. Our code is available at https://github.com/ihcho2/SAE-GPS.

Sparse autoencoders (SAEs) have emerged as a powerful analytical tool in mechanistic interpretability for large language models (LLMs), with growing success in applications beyond interpretability. Building on this momentum, we present a novel approach that leverages SAEs to enhance the general in-context learning (ICL) performance of LLMs.Specifically, we introduce Feature Detection through Prompt Variation (FDPV), which leverages the SAE’s remarkable ability to capture subtle differences between prompts, enabling efficient feature selection for downstream steering. In addition, we propose a novel steering method tailored to ICL—Selective In-Context Steering (SISTER)—grounded in recent insights from ICL research that LLMs utilize label words as key anchors. Our method yields a 3.5% average performance improvement across diverse text classification tasks and exhibits greater robustness to hyperparameter variations compared to standard steering approaches. Our code is available at https://github.com/ihcho2/SAE-ICL.

There has been a growing body of work focusing on the in-context learning (ICL) abilities of large language models (LLMs). However, it is an open question how effective ICL can be. This paper presents Tutor-ICL, a simple prompting method for classification tasks inspired by how effective instructors might engage their students in learning a task. Specifically, we propose presenting exemplar answers in a comparative format rather than the traditional single-answer format. We also show that including the test instance before the exemplars can improve performance, making it easier for LLMs to focus on relevant exemplars. Lastly, we include a summarization step before attempting the test, following a common human practice. Experiments on various classification tasks, conducted across both decoder-only LLMs (Llama 2, 3) and encoder-decoder LLMs (Flan-T5-XL, XXL), show that Tutor-ICL consistently boosts performance, achieving up to a 13.76% increase in accuracy.

We present a simple, but effective method to incorporate syntactic dependency information directly into transformer-based language models (e.g. RoBERTa) for tasks such as Aspect-Based Sentiment Classification (ABSC), where the desired output depends on specific input tokens. In contrast to prior approaches to ABSC that capture syntax by combining language models with graph neural networks over dependency trees, our model, Syntax-Integrated RoBERTa for ABSC (SIR-ABSC) incorporates syntax directly into the language model by using a novel aggregator token. Yet, SIR-ABSC outperforms these more complex models, yielding new state-of-the-art results on ABSC.