@inproceedings{bhowmik-etal-2026-script,
title = "Script Fragmentation and Format: What Drives the {E}nglish-{B}engali Performance Gap in Open {LLM}s?",
author = "Bhowmik, Shimanto and
Dipto, Tawsif Tashwar and
Islam, Md Sazzad and
Hsu, Sheryl and
Reasat, Tahsin",
editor = "Ponnusamy, Rahul and
Kumaresan, Prasanna Kumar and
Rajiakodi, Saranya and
B, Premjith and
MC, Shunmuga Priya and
Murugappan, Abirami and
G, Jyothish Lal and
V, Sowmya and
Chakravarthi, Bharathi Raja",
booktitle = "Proceedings of the Fifth Workshop on Multimodal Machine Learning in Low-resource Languages",
month = nov,
year = "2026",
address = "Hengqin, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.mmlow-main.1/",
pages = "1--17",
ISBN = "979-8-89176-445-3",
abstract = "Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across the task categories used to benchmark frontier models. We release 8 English benchmarks translated into Bengali with a single consistent pipeline and use them to evaluate 10 open LLMs from 4 families on paired English and Bengali inputs. Script fragmentation is what subword tokenizers do to Bengali{'}s alphasyllabary, whose written units are grapheme clusters spanning several Unicode code points: they cut the script into pieces smaller than a character, at a cost set by the vocabulary rather than the script itself. Format belongs to the evaluation, the answer shape that exact-match scoring demands regardless of whether the model knew the answer. Beyond confirming a substantial gap (macro LLM-judge score 0.79 in English versus 0.63 in Bengali), we show that part of it is a measurement artifact: exact-match accuracy conflates correctness with format adherence and because format failure is asymmetric across languages it distorts the apparent gap for some models three to five fold and even reverses its sign for one reasoning-tuned model. On the fragmentation side, Bengali costs roughly five times more tokens per word than English, the 10 models share only 4 vocabularies and that cost varies twofold across them (7.9 tokens per word under Llama3̃, 4.0 under Tekken), and under three of the four the average Bengali token spans fewer bytes than a single code point. Bengali is the constant here and the vocabularies are not, so what Bengali text costs is set by tokenizer design rather than by the script. Fertility and sequence length correlate only weakly with scores ($r = -0.23$), so we present this as a cost and segmentation concern rather than a driver of the gap. The datasets, pipeline and evaluation code are released."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bhowmik-etal-2026-script">
<titleInfo>
<title>Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?</title>
</titleInfo>
<name type="personal">
<namePart type="given">Shimanto</namePart>
<namePart type="family">Bhowmik</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tawsif</namePart>
<namePart type="given">Tashwar</namePart>
<namePart type="family">Dipto</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Md</namePart>
<namePart type="given">Sazzad</namePart>
<namePart type="family">Islam</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sheryl</namePart>
<namePart type="family">Hsu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tahsin</namePart>
<namePart type="family">Reasat</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-11</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the Fifth Workshop on Multimodal Machine Learning in Low-resource Languages</title>
</titleInfo>
<name type="personal">
<namePart type="given">Rahul</namePart>
<namePart type="family">Ponnusamy</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Prasanna</namePart>
<namePart type="given">Kumar</namePart>
<namePart type="family">Kumaresan</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Saranya</namePart>
<namePart type="family">Rajiakodi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Premjith</namePart>
<namePart type="family">B</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Shunmuga</namePart>
<namePart type="given">Priya</namePart>
<namePart type="family">MC</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Abirami</namePart>
<namePart type="family">Murugappan</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jyothish</namePart>
<namePart type="given">Lal</namePart>
<namePart type="family">G</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sowmya</namePart>
<namePart type="family">V</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Bharathi</namePart>
<namePart type="given">Raja</namePart>
<namePart type="family">Chakravarthi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Hengqin, China</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-445-3</identifier>
</relatedItem>
<abstract>Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across the task categories used to benchmark frontier models. We release 8 English benchmarks translated into Bengali with a single consistent pipeline and use them to evaluate 10 open LLMs from 4 families on paired English and Bengali inputs. Script fragmentation is what subword tokenizers do to Bengali’s alphasyllabary, whose written units are grapheme clusters spanning several Unicode code points: they cut the script into pieces smaller than a character, at a cost set by the vocabulary rather than the script itself. Format belongs to the evaluation, the answer shape that exact-match scoring demands regardless of whether the model knew the answer. Beyond confirming a substantial gap (macro LLM-judge score 0.79 in English versus 0.63 in Bengali), we show that part of it is a measurement artifact: exact-match accuracy conflates correctness with format adherence and because format failure is asymmetric across languages it distorts the apparent gap for some models three to five fold and even reverses its sign for one reasoning-tuned model. On the fragmentation side, Bengali costs roughly five times more tokens per word than English, the 10 models share only 4 vocabularies and that cost varies twofold across them (7.9 tokens per word under Llama3̃, 4.0 under Tekken), and under three of the four the average Bengali token spans fewer bytes than a single code point. Bengali is the constant here and the vocabularies are not, so what Bengali text costs is set by tokenizer design rather than by the script. Fertility and sequence length correlate only weakly with scores (r = -0.23), so we present this as a cost and segmentation concern rather than a driver of the gap. The datasets, pipeline and evaluation code are released.</abstract>
<identifier type="citekey">bhowmik-etal-2026-script</identifier>
<location>
<url>https://aclanthology.org/2026.mmlow-main.1/</url>
</location>
<part>
<date>2026-11</date>
<extent unit="page">
<start>1</start>
<end>17</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?
%A Bhowmik, Shimanto
%A Dipto, Tawsif Tashwar
%A Islam, Md Sazzad
%A Hsu, Sheryl
%A Reasat, Tahsin
%Y Ponnusamy, Rahul
%Y Kumaresan, Prasanna Kumar
%Y Rajiakodi, Saranya
%Y B, Premjith
%Y MC, Shunmuga Priya
%Y Murugappan, Abirami
%Y G, Jyothish Lal
%Y V, Sowmya
%Y Chakravarthi, Bharathi Raja
%S Proceedings of the Fifth Workshop on Multimodal Machine Learning in Low-resource Languages
%D 2026
%8 November
%I Association for Computational Linguistics
%C Hengqin, China
%@ 979-8-89176-445-3
%F bhowmik-etal-2026-script
%X Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across the task categories used to benchmark frontier models. We release 8 English benchmarks translated into Bengali with a single consistent pipeline and use them to evaluate 10 open LLMs from 4 families on paired English and Bengali inputs. Script fragmentation is what subword tokenizers do to Bengali’s alphasyllabary, whose written units are grapheme clusters spanning several Unicode code points: they cut the script into pieces smaller than a character, at a cost set by the vocabulary rather than the script itself. Format belongs to the evaluation, the answer shape that exact-match scoring demands regardless of whether the model knew the answer. Beyond confirming a substantial gap (macro LLM-judge score 0.79 in English versus 0.63 in Bengali), we show that part of it is a measurement artifact: exact-match accuracy conflates correctness with format adherence and because format failure is asymmetric across languages it distorts the apparent gap for some models three to five fold and even reverses its sign for one reasoning-tuned model. On the fragmentation side, Bengali costs roughly five times more tokens per word than English, the 10 models share only 4 vocabularies and that cost varies twofold across them (7.9 tokens per word under Llama3̃, 4.0 under Tekken), and under three of the four the average Bengali token spans fewer bytes than a single code point. Bengali is the constant here and the vocabularies are not, so what Bengali text costs is set by tokenizer design rather than by the script. Fertility and sequence length correlate only weakly with scores (r = -0.23), so we present this as a cost and segmentation concern rather than a driver of the gap. The datasets, pipeline and evaluation code are released.
%U https://aclanthology.org/2026.mmlow-main.1/
%P 1-17
Markdown (Informal)
[Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?](https://aclanthology.org/2026.mmlow-main.1/) (Bhowmik et al., MMLow 2026)
ACL