Transformer layers evolve representations through learned additive transformations that combine direction-preserving and direction-changing effects. We study this geometry by decomposing learned updates into parallel and perpendicular components. Across pretrained models, we find substantial learned parallel components, showing that direction-preserving behavior is not confined to the residual identity path. To test whether these components are redundant or behaviorally important, we apply the decomposition to residual-space updates and to value aggregation inside attention. Inference-time edits reveal a space-dependent asymmetry: value-space parallel edits preserve behavior more effectively than residual-space or perpendicular counterparts, especially when the direct self message is fixed and only the non-self aggregate is scaled. The decomposition also provides a compact, component-resolved view of compression error. During from-scratch pretraining, using parallel suppression as part of the forward computation lowers validation-loss trajectories and improves downstream averages in our evaluated runs, with the value-space variant strongest. Overall, this geometry connects update direction to editing robustness, compression diagnostics, and training-time intervention. Code is available in the https://github.com/Shwai-He/Transformer-Geometry.

Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations. We present a systematic analysis of this strategy: through spectral and geometric diagnostics, we show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. These findings suggest that token initialization is a key bottleneck when extending LMs with new vocabularies. Motivated by this diagnosis, we propose the **Grounded Token Initialization Hypothesis**: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains. We operationalize this hypothesis as **GTI** (Grounded Token Initialization), a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision. Despite its simplicity, **GTI** outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets. Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning, corroborating the hypothesis that initialization quality is a key bottleneck in vocabulary extension.