Inspired by Large Language Models (LLMs), codec-based TTS systems have achieved remarkable success in high-fidelity speech generation by directly adopting the standard Cross-Entropy (CE) objective to predict sequences of discrete speech tokens from neural audio codecs. However, this practice overlooks the geometric structure of codec token spaces: tokens that are close in the codec codebook embedding space often correspond to acoustically similar speech realizations and can serve as perceptually plausible alternatives. We term this observation the Neighboring Token Phenomenon. The one-hot CE objective is agnostic to such structure, as it penalizes all non-target tokens equally, including these close neighbors, creating a mismatch between the structured codec latent space and the learning objective. In this paper, we quantitatively analyze the neighboring token phenomenon in both reconstruction and generation settings. To address the mismatch, we propose Multi-Neighbor Cross-Entropy (MNCE) loss function, which generalizes the one-hot target of CE loss into a positive set containing the ground-truth token and its nearest neighbors, encouraging the model to treat neighbor predictions as partially correct. As a result, even when the model misses the exact target token, it is more likely to predict a close neighbor rather than an unrelated token, reducing perceptually harmful errors in synthesized speech. Experiments demonstrate that MNCE is effective and generalizable across diverse codec-based TTS systems, improving quality and robustness. Moreover, scaling training to 20,000 hours further validates the scalability of our approach.