Neural Cellular Automata (NCAs) generate complex spatial patterns in vision by repeatedly applying a shared local rule to persistent cell states, combining local perception, weight-shared iteration, and gated in-place updates. We test this principle in language modelling with TextNCA, a 1D causal model in which token positions act as cells, windowed attention provides local perception, and a GRU-style gate updates their states. Its hierarchical variant cascades increasingly wide attention windows w ∈ 8,32,128 and reuses each stage’s parameters for T_s iterations. We train roughly 30M-parameter models on SlimPajama for 60k steps and evaluate on WikiText-103. TextNCA trails a similarly sized Transformer-6L (60.3 versus 52.8 best-run PPL), so we study it as an analytical probe rather than a performance alternative. Reordering or flattening the windows raises PPL by 16.7–70.8, whereas a non-iterating sliding-window Transformer preserving the staged order performs comparably across three seeds. These results identify the narrow-to-wide schedule as the dominant factor and provide no reliable evidence that iterated weight sharing improves perplexity at this scale. Intermediate-state analyses support a prepare-then-decode account in which early stages reorganise representations and the final wide stage performs most next-token loss reduction.

Larger language models become simultaneously better and worse at handling contextual information—better at ignoring false claims, worse at ignoring irrelevant tokens. We formalize this apparent paradox through the first scaling laws for contextual entrainment, the tendency of models to favor tokens that appeared in context regardless of relevance. Analyzing the Cerebras-GPT (111M–13B) and Pythia (14M–12B) model families, we find entrainment follows predictable power-law scaling, but with opposite trends depending on context type: semantic contexts show decreasing entrainment with scale, while non-semantic contexts show increasing entrainment. Concretely, the largest models are four times more resistant to counterfactual misinformation than the smallest, yet simultaneously twice as prone to copying arbitrary tokens. These diverging trends, which replicate across model families, suggest that semantic filtering and mechanical copying are functionally distinct behaviors that scale in opposition. These opposing trends suggest that scaling alone does not resolve context sensitivity—it reshapes it.