Sparse Autoencoders (SAEs) decompose language model activations into sparse, interpretable features, but these features often struggle to predict task-relevant concepts that remain clearly encoded in the original dense activations. We call this mismatch a concept faithfulness gap: a faithful SAE should preserve concept-level representations, such as toxicity or sentiment, rather than producing features that look interpretable but lose predictive relevance. We introduce ProbeSAE, a probing-guided fine-tuning framework that addresses this gap by preserving task-relevant concept structure in a more monosemantic sparse decomposition, enabling more effective causal control over model behaviour without sacrificing interpretability. We evaluate ProbeSAE on two SAE architectures (TopK, JumpReLU), two domains (toxicity, sentiment), and multilingual out-of-distribution settings spanning six languages. Trained exclusively on English data, ProbeSAE: (i) produces more predictive, monosemantic, and interpretable sparse features than pre-trained or naïvely fine-tuned SAEs across all six evaluation languages; (ii) learns language-independent feature directions, as confirmed by cross-lingual causal ablation; (iii) passes SAEBench evaluation, confirming these gains do not arise from task/domain overfitting. Together, these results show that the concept faithfulness gap in SAEs can be addressed through probe-guided training, improving monosemanticity and causal effectiveness without sacrificing interpretability, making probe-guided SAEs a more reliable tool for domain-specific interpretability analysis.

Ensuring the safety of Large Language Models (LLMs) is a critical alignment challenge. Existing approaches often rely on invasive fine- tuning or external generation-based checks, which can be opaque and resource-inefficient. In this work, we investigate the geometry of safety concepts within pretrained representations, proposing a mechanistic methodology that identifies the layer where safe and unsafe concepts are maximally separable within a pretrained model’s representation space. By leveraging the intrinsic activation space of the optimal layer, we show that safety enforcement can be achieved via a simple linear classifier, avoiding the need for weight modification. We validate our framework across multiple domains (regulation, law, finance, cybersecurity, education, code, human resources, and social media), diverse tasks (safety classification, prompt injection, and toxicity detection), and 16 non-English languages on both encoder and decoder architectures. Our results show that: (i) the separation between safe and unsafe concepts emerges from a single layer direction in the activation space, (ii) monitoring internal representations provides a significantly more robust safeguarding mechanism compared to traditional evaluative or generative guardrail paradigms.

Interpreting the internal representations of large language models (LLMs) is crucial for their deployment in real-world applications, impacting areas such as AI safety, debugging, and compliance. Sparse Autoencoders facilitate interpretability by decomposing polysemantic activation into a latent space of monosemantic features. However, evaluating the auto-interpretability of these features is difficult and computationally expensive, which limits scalability in practical settings. In this work, we propose SFAL, an alternative evaluation strategy that reduces reliance on LLM-based scoring by assessing the alignment between the semantic neighbourhoods of features (derived from auto-interpretation embeddings) and their functional neighbourhoods (derived from co-occurrence statistics).Our method enhances efficiency, enabling fast and cost-effective assessments. We validate our approach on large-scale models, demonstrating its potential to provide interpretability while reducing computational overhead, making it suitable for real-world deployment.

We present BEEP (BEst DrivEr’s License Performer), a benchmark challenge to evaluate large language models in the context of a simulated Italian driver’s license exam. This challenge tests the models’ ability to understand and apply traffic laws, road safety regulations, and vehicle-related knowledge through a series of true/false questions. The dataset is derived from official ministerial materials used in the Italian licensing process, specifically targeting Category B licenses.We evaluate models such as LLaMA and Mixtral across multiple categories. In addition, we simulate a driving license test to assess the models’ real-world applicability, where the pass rate is determined based on the number of errors allowed. While scaling up model size improved performance, even larger models struggled to pass the exam consistently. The challenge demonstrates the capabilities and limitations of LLMs in handling real-world, high-stakes scenarios, providing insights into their practical use and areas for further improvement.