@inproceedings{al-zubayer-etal-2026-sink,
title = "Same Sink, Different Plumbing: Reproducing and Refining the Mechanism of Attention Sinks",
author = "Al Zubayer, Abdullah and
Mahmood, Syed Naveed and
Bhuiyan, Md Rezaur Rahman and
Abrar, Mir Md Mohiuddin and
Khondaker, Jareen Tasneem and
Zaman, Tasfia and
Sadeque, Farig",
editor = "Arad, Dana and
Belinkov, Yonatan and
Chen, Hanjie and
Feldhus, Nils and
Mueller, Aaron and
Mohebbi, Hosein and
Sarti, Gabriele and
Tutek, Martin",
booktitle = "Proceedings of the 9th {B}lackbox{NLP} Workshop: Analyzing and Interpreting Neural Networks for {NLP}",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.blackboxnlp-1.71/",
pages = "1233--1247",
ISBN = "979-8-89176-476-7",
abstract = "Attention sinks assign large, largely content-independent attention to the first position. Ran-Milo et al. (2026) explain GPT-2{'}s sink with a parameter-level circuit: a learned query bias $\mathbf{b}_Q$ aligns with the key projection of the first position{'}s effective positional encoding, whose few massive coordinates amplify the score. They infer that other architectures build sinks differently but do not measure this claim. We reproduce the original intervention effects and stress-test their causal interpretation. Baseline plus nine interventions and four structural analyses reproduce under the original evaluation, with two scope qualifications; the massive coordinates are exactly {138, 378, 447}. Yet nullifying $\mathbf{b}_Q$ leaves 44.5{\%} of the sink, which follows an activation-dependent query route into the same first-position key. This *anchor*{--}*route* distinction unifies the extensions: with GPT-2 scale, the bias route fades while the activation-dependent route concentrates into fewer carrier heads; across 16 checkpoints in four families, no qualitative profile over mutually available conditions spans all families; and across five pre-training runs, anchoring recurs despite unstable coordinate identities. In GPT-2-small at 40 tokens, Swap EPE incurs about one quarter of the cross-entropy cost of Remove First PE. The stable phenomenon is an anchoring computation, not a universal parameter-level circuit, so mitigation must be validated against the target model{'}s wiring."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="al-zubayer-etal-2026-sink">
<titleInfo>
<title>Same Sink, Different Plumbing: Reproducing and Refining the Mechanism of Attention Sinks</title>
</titleInfo>
<name type="personal">
<namePart type="given">Abdullah</namePart>
<namePart type="family">Al Zubayer</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Syed</namePart>
<namePart type="given">Naveed</namePart>
<namePart type="family">Mahmood</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Md</namePart>
<namePart type="given">Rezaur</namePart>
<namePart type="given">Rahman</namePart>
<namePart type="family">Bhuiyan</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Mir</namePart>
<namePart type="given">Md</namePart>
<namePart type="given">Mohiuddin</namePart>
<namePart type="family">Abrar</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jareen</namePart>
<namePart type="given">Tasneem</namePart>
<namePart type="family">Khondaker</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tasfia</namePart>
<namePart type="family">Zaman</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Farig</namePart>
<namePart type="family">Sadeque</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP</title>
</titleInfo>
<name type="personal">
<namePart type="given">Dana</namePart>
<namePart type="family">Arad</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yonatan</namePart>
<namePart type="family">Belinkov</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Hanjie</namePart>
<namePart type="family">Chen</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Nils</namePart>
<namePart type="family">Feldhus</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Aaron</namePart>
<namePart type="family">Mueller</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Hosein</namePart>
<namePart type="family">Mohebbi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Gabriele</namePart>
<namePart type="family">Sarti</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Martin</namePart>
<namePart type="family">Tutek</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-476-7</identifier>
</relatedItem>
<abstract>Attention sinks assign large, largely content-independent attention to the first position. Ran-Milo et al. (2026) explain GPT-2’s sink with a parameter-level circuit: a learned query bias \mathbfb_Q aligns with the key projection of the first position’s effective positional encoding, whose few massive coordinates amplify the score. They infer that other architectures build sinks differently but do not measure this claim. We reproduce the original intervention effects and stress-test their causal interpretation. Baseline plus nine interventions and four structural analyses reproduce under the original evaluation, with two scope qualifications; the massive coordinates are exactly 138, 378, 447. Yet nullifying \mathbfb_Q leaves 44.5% of the sink, which follows an activation-dependent query route into the same first-position key. This *anchor*–*route* distinction unifies the extensions: with GPT-2 scale, the bias route fades while the activation-dependent route concentrates into fewer carrier heads; across 16 checkpoints in four families, no qualitative profile over mutually available conditions spans all families; and across five pre-training runs, anchoring recurs despite unstable coordinate identities. In GPT-2-small at 40 tokens, Swap EPE incurs about one quarter of the cross-entropy cost of Remove First PE. The stable phenomenon is an anchoring computation, not a universal parameter-level circuit, so mitigation must be validated against the target model’s wiring.</abstract>
<identifier type="citekey">al-zubayer-etal-2026-sink</identifier>
<location>
<url>https://aclanthology.org/2026.blackboxnlp-1.71/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>1233</start>
<end>1247</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Same Sink, Different Plumbing: Reproducing and Refining the Mechanism of Attention Sinks
%A Al Zubayer, Abdullah
%A Mahmood, Syed Naveed
%A Bhuiyan, Md Rezaur Rahman
%A Abrar, Mir Md Mohiuddin
%A Khondaker, Jareen Tasneem
%A Zaman, Tasfia
%A Sadeque, Farig
%Y Arad, Dana
%Y Belinkov, Yonatan
%Y Chen, Hanjie
%Y Feldhus, Nils
%Y Mueller, Aaron
%Y Mohebbi, Hosein
%Y Sarti, Gabriele
%Y Tutek, Martin
%S Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-476-7
%F al-zubayer-etal-2026-sink
%X Attention sinks assign large, largely content-independent attention to the first position. Ran-Milo et al. (2026) explain GPT-2’s sink with a parameter-level circuit: a learned query bias \mathbfb_Q aligns with the key projection of the first position’s effective positional encoding, whose few massive coordinates amplify the score. They infer that other architectures build sinks differently but do not measure this claim. We reproduce the original intervention effects and stress-test their causal interpretation. Baseline plus nine interventions and four structural analyses reproduce under the original evaluation, with two scope qualifications; the massive coordinates are exactly 138, 378, 447. Yet nullifying \mathbfb_Q leaves 44.5% of the sink, which follows an activation-dependent query route into the same first-position key. This *anchor*–*route* distinction unifies the extensions: with GPT-2 scale, the bias route fades while the activation-dependent route concentrates into fewer carrier heads; across 16 checkpoints in four families, no qualitative profile over mutually available conditions spans all families; and across five pre-training runs, anchoring recurs despite unstable coordinate identities. In GPT-2-small at 40 tokens, Swap EPE incurs about one quarter of the cross-entropy cost of Remove First PE. The stable phenomenon is an anchoring computation, not a universal parameter-level circuit, so mitigation must be validated against the target model’s wiring.
%U https://aclanthology.org/2026.blackboxnlp-1.71/
%P 1233-1247
Markdown (Informal)
[Same Sink, Different Plumbing: Reproducing and Refining the Mechanism of Attention Sinks](https://aclanthology.org/2026.blackboxnlp-1.71/) (Al Zubayer et al., BlackboxNLP 2026)
ACL
- Abdullah Al Zubayer, Syed Naveed Mahmood, Md Rezaur Rahman Bhuiyan, Mir Md Mohiuddin Abrar, Jareen Tasneem Khondaker, Tasfia Zaman, and Farig Sadeque. 2026. Same Sink, Different Plumbing: Reproducing and Refining the Mechanism of Attention Sinks. In Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 1233–1247, Budapest, Hungary. Association for Computational Linguistics.