Large language models (LLMs) encode a wide range of information about the world, but understanding how such knowledge is represented remains a central challenge in mechanistic interpretability. In this work, we reproduce and extend a recent study investigating the representation of geographic information in LLMs using sparse autoencoders (SAEs). We focus on this work because, to the best of our knowledge, it is the only study that explicitly examines geographic representations in LLMs through the lens of feature-based interpretability. While the original study identifies SAE features that appear to encode geographic concepts, we extend the analysis to a broader set of locations spanning the globe and investigate the robustness of these findings. In particular, we hypothesize that some of the identified features may not represent geographic concepts directly, but may instead be driven by lexical properties of placenames, such as highly discriminative character sequences or syllables. To test this hypothesis, we introduce systematic perturbations of placenames that preserve their geographic reference while altering their orthographic form, and examine the resulting changes in SAE activation patterns and geographic activation maps. Our results provide a broader assessment of the reproducibility and generalizability of geographic representations identified by the original study, while highlighting potential confounds that should be considered when interpreting SAE features as semantic or conceptual representations. We position this work within the Generalizability and Baselines themes of the Reproducibility Challenge: we test whether the original findings hold beyond their initial experimental scope, and we introduce a simple lexical baseline that recovers most of the reported geographic signal from surface form alone, without access to the model’s internal representations.

Nous analysons l’impact de plusieurs facteurs d’optimisation sur la qualité des informations géographiques contenues dans des grands modèles de langue (LLMs) : famille, taille, «quantization», «instruction fine-tuning», prompt et langue. Nous évaluons également la qualité des représentations internes, en particulier pour les modèles génératifs ayant des difficultés à suivre les instructions. Nos résultats montrent que la quantization dégrade nettement les performances, tandis que les versions conversationnelles («Instruct») perdent généralement en qualité d’informations par rapport à leur version «base», à l’exception des modèles de petite taille. L’ensemble de notre protocole d’évaluation est entièrement reproductible et disponible en accès libre.