Abdullah Al Zubayer

Author directory

2026

Attention sinks assign large, largely content-independent attention to the first position. Ran-Milo et al. (2026) explain GPT-2’s sink with a parameter-level circuit: a learned query bias 𝐛Q aligns with the key projection of the first position’s effective positional encoding, whose few massive coordinates amplify the score. They infer that other architectures build sinks differently but do not measure this claim. We reproduce the original intervention effects and stress-test their causal interpretation. Baseline plus nine interventions and four structural analyses reproduce under the original evaluation, with two scope qualifications; the massive coordinates are exactly 138, 378, 447. Yet nullifying 𝐛Q leaves 44.5% of the sink, which follows an activation-dependent query route into the same first-position key. This *anchor*–*route* distinction unifies the extensions: with GPT-2 scale, the bias route fades while the activation-dependent route concentrates into fewer carrier heads; across 16 checkpoints in four families, no qualitative profile over mutually available conditions spans all families; and across five pre-training runs, anchoring recurs despite unstable coordinate identities. In GPT-2-small at 40 tokens, Swap EPE incurs about one quarter of the cross-entropy cost of Remove First PE. The stable phenomenon is an anchoring computation, not a universal parameter-level circuit, so mitigation must be validated against the target model’s wiring.