Gianfranco Lombardo

Author directory

2026

Lens methods aim to decode intermediate representations of large language models into vocabulary predictions by training independent translators for each layer. While this approach provides insights into how the final response emerges across layers, it overlooks the dominant structure shared among them. We identify that per-layer alignment problems share a dominant common structure: the first singular component of per-layer translators explains 31–64% of total spectral variance, 10–20× above the unstructured baseline. Motivated by this evidence, we propose AnchorLens, which exploits a global-local decomposition: a shared full-rank matrix G maps layers into a common coordinate system, while per-layer low-rank adapters correct layer-specific residuals, trained in two stages. AnchorLens achieves over 90% parameter reduction relative to full-rank methods while matching TunedLens and outperforming LoRALens on next-token prediction and linear shortcut evaluations across eight benchmark tasks. These results hold across 16 models up to 120B parameters from five families: GPT-OSS, Qwen3, Qwen2.5, OLMo2, and Gemma2. Furthermore, the shared coordinate system induced by G enables geometric analyses inaccessible to independent per-layer methods, including quantification of inter-layer drift reduction and empirical validation of non-degeneracy: post-training adapter orthogonality confirms G absorbs the shared structure. Our code is publicly available on https://github.com/GiuseT/InterpretabilityLenses.git.