@inproceedings{dejonge-chambers-2026-llms,
title = "{LLM}s are Easily Startled: Fragility Testing with Lookalike Character Noise",
author = "deJonge, Peter and
Chambers, Nathanael",
editor = "Bak, JinYeong and
van der Goot, Rob and
Jang, Hyeju and
Buaphet, Weerayut and
Ramponi, Alan and
Xu, Wei and
Ritter, Alan",
booktitle = "Proceedings of the 11th Workshop on Natural User-generated Text",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.wnut-1.12/",
pages = "167--181",
ISBN = "979-8-89176-470-5",
abstract = "Language models are dependent on the character sets of the languages seen during training. Some languages use large character sets, like Chinese and Korean, while others are very small, like English. When the input during inference diverges from the trained character set, it can lead to unexpected behavior. While many researchers have looked at using this weakness for adversarial attacks, these studies typically focus on extreme cases that lead to model failure. But we don{'}t have a strong understanding of how brittle models are to more casual and routine character substitutions. In fact, many `known' characters have Unicode lookalikes that users enjoy using without malicious purposes. This paper studies the fragility of models when faced with such character noise. When input contains innocuous character substitutions, do models robustly handle the distortion? How does the tokenizer and overall model size affect performance? This paper presents analyses of tokenizers and a series of fragility tests to answer these questions. We find that tokenization is not the only stumbling block, and neither is a larger model size the assured solution. We present a formal benchmark, publicly release a Fragility Scorer, and describe our experimental results on a variety of open-source models. Implications and suggestions for improving fragility are then presented."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="dejonge-chambers-2026-llms">
<titleInfo>
<title>LLMs are Easily Startled: Fragility Testing with Lookalike Character Noise</title>
</titleInfo>
<name type="personal">
<namePart type="given">Peter</namePart>
<namePart type="family">deJonge</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Nathanael</namePart>
<namePart type="family">Chambers</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 11th Workshop on Natural User-generated Text</title>
</titleInfo>
<name type="personal">
<namePart type="given">JinYeong</namePart>
<namePart type="family">Bak</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Rob</namePart>
<namePart type="family">van der Goot</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Hyeju</namePart>
<namePart type="family">Jang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Weerayut</namePart>
<namePart type="family">Buaphet</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Alan</namePart>
<namePart type="family">Ramponi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Wei</namePart>
<namePart type="family">Xu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Alan</namePart>
<namePart type="family">Ritter</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-470-5</identifier>
</relatedItem>
<abstract>Language models are dependent on the character sets of the languages seen during training. Some languages use large character sets, like Chinese and Korean, while others are very small, like English. When the input during inference diverges from the trained character set, it can lead to unexpected behavior. While many researchers have looked at using this weakness for adversarial attacks, these studies typically focus on extreme cases that lead to model failure. But we don’t have a strong understanding of how brittle models are to more casual and routine character substitutions. In fact, many ‘known’ characters have Unicode lookalikes that users enjoy using without malicious purposes. This paper studies the fragility of models when faced with such character noise. When input contains innocuous character substitutions, do models robustly handle the distortion? How does the tokenizer and overall model size affect performance? This paper presents analyses of tokenizers and a series of fragility tests to answer these questions. We find that tokenization is not the only stumbling block, and neither is a larger model size the assured solution. We present a formal benchmark, publicly release a Fragility Scorer, and describe our experimental results on a variety of open-source models. Implications and suggestions for improving fragility are then presented.</abstract>
<identifier type="citekey">dejonge-chambers-2026-llms</identifier>
<location>
<url>https://aclanthology.org/2026.wnut-1.12/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>167</start>
<end>181</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T LLMs are Easily Startled: Fragility Testing with Lookalike Character Noise
%A deJonge, Peter
%A Chambers, Nathanael
%Y Bak, JinYeong
%Y van der Goot, Rob
%Y Jang, Hyeju
%Y Buaphet, Weerayut
%Y Ramponi, Alan
%Y Xu, Wei
%Y Ritter, Alan
%S Proceedings of the 11th Workshop on Natural User-generated Text
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-470-5
%F dejonge-chambers-2026-llms
%X Language models are dependent on the character sets of the languages seen during training. Some languages use large character sets, like Chinese and Korean, while others are very small, like English. When the input during inference diverges from the trained character set, it can lead to unexpected behavior. While many researchers have looked at using this weakness for adversarial attacks, these studies typically focus on extreme cases that lead to model failure. But we don’t have a strong understanding of how brittle models are to more casual and routine character substitutions. In fact, many ‘known’ characters have Unicode lookalikes that users enjoy using without malicious purposes. This paper studies the fragility of models when faced with such character noise. When input contains innocuous character substitutions, do models robustly handle the distortion? How does the tokenizer and overall model size affect performance? This paper presents analyses of tokenizers and a series of fragility tests to answer these questions. We find that tokenization is not the only stumbling block, and neither is a larger model size the assured solution. We present a formal benchmark, publicly release a Fragility Scorer, and describe our experimental results on a variety of open-source models. Implications and suggestions for improving fragility are then presented.
%U https://aclanthology.org/2026.wnut-1.12/
%P 167-181
Markdown (Informal)
[LLMs are Easily Startled: Fragility Testing with Lookalike Character Noise](https://aclanthology.org/2026.wnut-1.12/) (deJonge & Chambers, WNUT 2026)
ACL