Language models are dependent on the character sets of the languages seen during training. Some languages use large character sets, like Chinese and Korean, while others are very small, like English. When the input during inference diverges from the trained character set, it can lead to unexpected behavior. While many researchers have looked at using this weakness for adversarial attacks, these studies typically focus on extreme cases that lead to model failure. But we don’t have a strong understanding of how brittle models are to more casual and routine character substitutions. In fact, many ‘known’ characters have Unicode lookalikes that users enjoy using without malicious purposes. This paper studies the fragility of models when faced with such character noise. When input contains innocuous character substitutions, do models robustly handle the distortion? How does the tokenizer and overall model size affect performance? This paper presents analyses of tokenizers and a series of fragility tests to answer these questions. We find that tokenization is not the only stumbling block, and neither is a larger model size the assured solution. We present a formal benchmark, publicly release a Fragility Scorer, and describe our experimental results on a variety of open-source models. Implications and suggestions for improving fragility are then presented.