Giuseppe Trimigno
Author directory2026
AnchorLens: A Shared Coordinate System for Interpreting Intermediate Representations in Large Language Models
Giuseppe Trimigno | Gianfranco Lombardo | Stefano Cagnoni
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Giuseppe Trimigno | Gianfranco Lombardo | Stefano Cagnoni
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Lens methods aim to decode intermediate representations of large language models into vocabulary predictions by training independent translators for each layer. While this approach provides insights into how the final response emerges across layers, it overlooks the dominant structure shared among them. We identify that per-layer alignment problems share a dominant common structure: the first singular component of per-layer translators explains 31–64% of total spectral variance, 10–20× above the unstructured baseline. Motivated by this evidence, we propose AnchorLens, which exploits a global-local decomposition: a shared full-rank matrix G maps layers into a common coordinate system, while per-layer low-rank adapters correct layer-specific residuals, trained in two stages. AnchorLens achieves over 90% parameter reduction relative to full-rank methods while matching TunedLens and outperforming LoRALens on next-token prediction and linear shortcut evaluations across eight benchmark tasks. These results hold across 16 models up to 120B parameters from five families: GPT-OSS, Qwen3, Qwen2.5, OLMo2, and Gemma2. Furthermore, the shared coordinate system induced by G enables geometric analyses inaccessible to independent per-layer methods, including quantification of inter-layer drift reduction and empirical validation of non-degeneracy: post-training adapter orthogonality confirms G absorbs the shared structure. Our code is publicly available on https://github.com/GiuseT/InterpretabilityLenses.git.