Hyunil Kim
Author directory2026
Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs
Seunghan Kim | Minyeong Choe | Hyunil Kim | Haehyun Cho
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Seunghan Kim | Minyeong Choe | Hyunil Kim | Haehyun Cho
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Multilingual LLMs answer the same multi-hop reasoning question across languages, but we lack a mechanistic account of whether they share an internal circuit. We identify Bridge Routing Heads (BRH) in two large multilingual LLMs through a three-stage pipeline. The resulting language-specific head sets exhibit near-complete mutual exclusivity across the five languages, with a mean Jaccard similarity of only 0.017 for Llama 3.1 70B and 0.057 for Qwen 2.5 72B, revealing language-idiosyncratic circuits. Ablating general BRH increases two-hop Negative Log-Likelihood (NLL) by 39-89× the random-head baseline, providing direct causal evidence of their role. Amplifying these heads in a failing target-language pass rescues up to 51.7% of cross-lingual failures, with no training. The two models share this dual-circuit pattern but allocate heads differently: Llama concentrates chaining in a large general pool, while Qwen leans on larger language-specific pools. Together these results show that activation-level intervention alone can recover correct answers from cross-lingual reasoning failures.
2025
Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanisms
Minyeong Choe | Haehyun Cho | Changho Seo | Hyunil Kim
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Minyeong Choe | Haehyun Cho | Changho Seo | Hyunil Kim
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Understanding how Transformer-based language models store and retrieve factual associations is critical for improving interpretability and enabling targeted model editing. Prior work, primarily on GPT-style models, has identified MLP modules in early layers as key contributors to factual recall. However, it remains unclear whether these findings generalize across different autoregressive architectures. To address this, we conduct a comprehensive evaluation of factual recall across several models—including GPT, LLaMA, Qwen, and DeepSeek—analyzing where and how factual information is encoded and accessed. Consequently, we find that Qwen-based models behave differently from previous patterns: attention modules in the earliest layers contribute more to factual recall than MLP modules. Our findings suggest that even within the autoregressive Transformer family, architectural variations can lead to fundamentally different mechanisms of factual recall.