Munachiso S Nwadike
Author directoryAlso published as: Munachiso Samuel Nwadike
2026
Sparse Long Memory and Transition Bottlenecks Shape Positional Recall in Mamba Language Models
Muhammad Cendekia Airlangga | Hilal AlQuabeh | Munachiso Samuel Nwadike | Kentaro Inui
Findings of the Association for Computational Linguistics: EMNLP 2026
Muhammad Cendekia Airlangga | Hilal AlQuabeh | Munachiso Samuel Nwadike | Kentaro Inui
Findings of the Association for Computational Linguistics: EMNLP 2026
Mamba language models process sequences recurrently, but recurrent processing does not imply that all tokens are retained equally. We study this gap using structured recall tasks that expose primacy, recency, and lost-in-the-middle behavior. Our results suggest that Mamba uses two functional memory regimes: a short-lived state that supports recency, and a sparse long-lived channel subspace that preserves selected early inputs. Targeted interventions on this long-memory subspace selectively degrade early recall and transfer to long-context QA and summarization, indicating that the same channels support real long-context use. We then ask why middle items are forgotten despite being processed sequentially. We find that the central failure is a short-to-long memory transition bottleneck: middle items are no longer recent enough for short memory, but are often not promoted into the sparse long-lived subspace. Repeated input structure strengthens this bottleneck by changing the input-dependent discretization dynamics. These findings provide a mechanistic account of Mamba’s U-shaped recall curve as the interaction between long memory, short memory, and a fragile transition between them.
Sycophancy Hides Linearly in the Attention Heads
Rifo Ahmad Genadi | Munachiso Samuel Nwadike | Nurdaulet Mukhituly | Tatsuya Hiraoka | Hilal AlQuabeh | Kentaro Inui
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Rifo Ahmad Genadi | Munachiso Samuel Nwadike | Nurdaulet Mukhituly | Tatsuya Hiraoka | Hilal AlQuabeh | Kentaro Inui
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
We find that correct-to-incorrect sycophancy signals are most linearly accessible within multi-head attention activations. Motivated by the linear representation hypothesis, we train linear probes across the residual stream, multilayer perceptron (MLP), and attention layers to analyze where these signals emerge. Although separability appears in the residual stream and MLPs, steering using these probes is most effective in a sparse subset of middle-layer attention heads. Using TruthfulQA as the base dataset, we find that probes trained on it transfer effectively to other factual QA benchmarks. Furthermore, comparing our discovered direction to previously identified “truthful” directions reveals limited overlap, suggesting that factual accuracy, and deference resistance, arise from related but distinct mechanisms. Attention-pattern analysis further indicates that the influential heads attend disproportionately to expressions of user doubt, contributing to sycophantic shifts. Overall, these findings suggest that sycophancy can be mitigated through simple, targeted linear interventions that exploit the internal geometry of attention activations. Code will be released upon publication.
Mitigating the Impact of Glitch Tokens via Targeted Retokenization
Muhammad Alif Al Hakim | Dmitrii Kuzmin | Carlos Javier Galán Zavala | Tatsuya Hiraoka | Munachiso Samuel Nwadike | Ayana Niwa | Marco Cognetta | David Pohl | Kentaro Inui
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Muhammad Alif Al Hakim | Dmitrii Kuzmin | Carlos Javier Galán Zavala | Tatsuya Hiraoka | Munachiso Samuel Nwadike | Ayana Niwa | Marco Cognetta | David Pohl | Kentaro Inui
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
This paper examines whether avoiding undertrained or “glitch” tokens and maximizing tokenization quality during inference can improve downstream task performance without retraining the model. We evaluate this question using a retokenization method that selects the highest-quality tokenization based on the trainedness of tokens, which is defined through the norm or entropy of token embeddings. We compare the performance difference between the tokenization that includes undertrained tokens and the one that avoids such tokens on multiple benchmarks. Our findings reveal a key nuance: undertrained tokens degrade performance primarily when they occupy semantically salient positions that are critical for solving the task. Consequently, retokenization preserves performance on ordinary inputs, while yielding consistent performance improvements when such tokens appear in high-importance contexts.
2025
Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles
Munachiso S Nwadike | Zangir Iklassov | Toluwani Aremu | Tatsuya Hiraoka | Benjamin Heinzerling | Velibor Bojkovic | Hilal AlQuabeh | Martin Takáč | Kentaro Inui
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Munachiso S Nwadike | Zangir Iklassov | Toluwani Aremu | Tatsuya Hiraoka | Benjamin Heinzerling | Velibor Bojkovic | Hilal AlQuabeh | Martin Takáč | Kentaro Inui
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
We introduce the concept of the self-referencing causal cycle (abbreviated ReCall )—a mechanism that enables large language models (LLMs) to bypass the limitations of unidirectional causality, which underlies a phenomenon known as the reversal curse. When an LLM is prompted with sequential data, it often fails to recall preceding context. For example, when we ask an LLM to recall the line preceding “O say does that star-spangled banner yet wave” in the U.S. National Anthem, it often fails to correctly return “Gave proof through the night that our flag was still there”—this is due to the reversal curse. It occurs because language models such as ChatGPT and Llama generate text based on preceding tokens, requiring facts to be learned and reproduced in a consistent token order. While the reversal curse is often viewed as a limitation, we offer evidence of an alternative view: it is not always an obstacle in practice. We find that ReCall is driven by what we designate as cycle tokens—sequences that connect different parts of the training data, enabling recall of preceding tokens from succeeding ones. Through rigorous probabilistic formalization and controlled experiments, we demonstrate how the cycles they induce influence a model’s ability to reproduce information. To facilitate reproducibility, we provide our code and experimental details at https://anonymous.4open.science/r/remember-B0B8/.
Search