This paper examines whether avoiding undertrained or “glitch” tokens and maximizing tokenization quality during inference can improve downstream task performance without retraining the model. We evaluate this question using a retokenization method that selects the highest-quality tokenization based on the trainedness of tokens, which is defined through the norm or entropy of token embeddings. We compare the performance difference between the tokenization that includes undertrained tokens and the one that avoids such tokens on multiple benchmarks. Our findings reveal a key nuance: undertrained tokens degrade performance primarily when they occupy semantically salient positions that are critical for solving the task. Consequently, retokenization preserves performance on ordinary inputs, while yielding consistent performance improvements when such tokens appear in high-importance contexts.