ChunXia Yang

Author directory

2026

External hidden-state memories can extend Transformers beyond local attention windows, but the write policy deciding which states enter finite memory is often treated as an implementation detail. We study this factor with a controlled delayed-copy diagnostic using a windowed Transformer augmented with an external FIFO hidden-state memory. Under forced masking, query-to-source attention is blocked, removing this shortcut and making source retention directly testable. We define a capacity boundary as the minimum memory size needed to reach a fixed copy-accuracy threshold. We find that write policy changes this boundary predictably under constrained FIFO memory: an idealized source-only policy, which assumes that relevant states are identifiable at write time, yields an approximately delay-invariant boundary near the source length, whereas an unfiltered prefix-all policy induces a delay-dependent boundary that grows with written non-source content. Intermediate partial-prefix write policies produce intermediate boundaries, while source-pinned controls and memory diagnostics show that prefix-all degradation is dominated by FIFO eviction of source states rather than retrieval interference from retained non-source states alone. A symbolic key–value retrieval diagnostic shows the same qualitative trend. Within this controlled setting, our results highlight write policy as a central variable in evaluating finite FIFO external-memory Transformers.