The functional organization of prosody within the Residual Vector Quantization (RVQ) codebooks of neural audio codecs remains an open question. Prior work relies on correlational probing, showing that pitch features can be decoded from frozen codebook embeddings. Decodability, however, does not imply causal necessity: a codebook may encode pitch information the decoder ignores, while another may be essential despite containing little probe-recoverable signal. We address this gap through causal output-side ablation: for each codebook, we zero or mean-replace its discrete tokens, re-decode the waveform, and measure degradation in fundamental frequency F₀ (CREPE), energy (frame-wise RMS), and intelligibility (STOI) against the unablated reconstruction. Experiments span 200 LibriSpeech test-clean utterances under clean, 15, 5, and 0 dB SNR with MUSAN noise across multiple neural audio codecs. Across different architectures, we find a consistent two-tier causal structure: a codec-specific subset of codebooks is reconstruction-critical, while the remaining intelligibility-preserving codebooks exhibit a stable pitch-contribution hierarchy that persists under additive noise. These findings provide new evidence for how neural audio codecs causally organize prosodic information.

Outlier dimensions are a recurring geometric property of transformer representations that have been linked to token frequency in pretrained language models. However, studies on pretrained models cannot disentangle token frequency from semantic and optimization confounds, preventing a causal understanding of whether frequency itself drives outlier formation. These distortions also degrade probing accuracy and hinder model compression, motivating a controlled study of their origin. We address this issue by isolating frequency skew as a controlled variable in a synthetic transformer trained on data with a tunable Zipfian token distribution. We find that outlier magnitude remains stable under low skew but increases sharply beyond a moderate threshold. A frozen frequency-based output bias fails to reproduce this behavior, model capacity exhibits a non-monotonic U-shaped effect, and an L₂-normalized cosine softmax suppresses outlier geometry only under a large temperature not reached by standard training. These results identify frequency skew as a causal driver of outlier dimensions and show that removing their geometry requires an intervention beyond standard training.