The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding. Despite its theoretical elegance, its implementation in practice is complex, limiting its adoption to the SentencePiece package and adapters thereof. We bridge this gap between theory and practice by providing a clear guide to implementation and parameter choices. We also identify a simpler algorithm that accepts slightly higher training loss in exchange for improved compression.

LLM benchmark labels are frozen at release and silently copied into downstream benchmarks, errors and all. Our Item Response Theory indicator surfaces these mislabels at 95% precision in the top 200, across 114 models on seven preference and multiple-choice benchmarks, outperforming a supervised classifier. We trace these errors to mechanical labeling rules, upstream errors inherited unchanged from source datasets, and ambiguous items with no defensible single key. The same fit reveals that reward models specialize in stylistic preference over factual knowledge, and exposes one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with contamination or benchmark-specific over-optimization.

The disconnect between tokenizer creation and model training in language models allows for specific inputs, such as the infamous SolidGoldMagikarp token, to induce unwanted model behaviour. Although such ‘glitch tokens’, tokens present in the tokenizer vocabulary but that are nearly or entirely absent during model training, have been observed across various models, a reliable method to identify and address them has been missing. We present a comprehensive analysis of Large Language Model tokenizers, specifically targeting this issue of detecting under-trained tokens. Through a combination of tokenizer analysis, model weight-based indicators, and prompting techniques, we develop novel and effective methods for automatically detecting these problematic tokens. Our findings demonstrate the prevalence of such tokens across a diverse set of models and provide insights into improving the efficiency and safety of language models.