Lost Without Translation? Multilingual Sentence Embeddings for Linguistic-integrated Reliability Auditing

Ummugul Bezirhan, Ji Yoon Jung, Matthias von Davier


Abstract
Multilingual assessment systems commonly rely on translation for scoring and quality-control processes. We evaluate whether multilingual sentence embeddings can replace translated English input for Linguistic-Integrated Reliability Auditing (LiRA) across 11 PIRLS constructed-response items and three embedding models. Native-language embeddings reproduced translation-based reliability estimates closely while recovering responses excluded after translation failure, with no meaningful change in reliability.
Anthology ID:
2026.aimecon-main.49
Volume:
Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers
Month:
October
Year:
2026
Address:
Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States
Editors:
Joshua Wilson, Christopher Ormerod, Magdalen Beiting-Parrish
Venue:
AIME-Con
SIG:
Publisher:
National Council on Measurement in Education (NCME)
Note:
Pages:
443–449
Language:
URL:
https://aclanthology.org/2026.aimecon-main.49/
DOI:
Bibkey:
Cite (ACL):
Ummugul Bezirhan, Ji Yoon Jung, and Matthias von Davier. 2026. Lost Without Translation? Multilingual Sentence Embeddings for Linguistic-integrated Reliability Auditing. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers, pages 443–449, Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).
Cite (Informal):
Lost Without Translation? Multilingual Sentence Embeddings for Linguistic-integrated Reliability Auditing (Bezirhan et al., AIME-Con 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.aimecon-main.49.pdf