Leading Whitespaces of Language Models’ Subword Vocabulary Pose a Confound for Calculating Word Probabilities

Leading Whitespaces of Language Models’ Subword Vocabulary Pose a Confound for Calculating Word Probabilities Byung-Doh Oh author William Schuler author 2024-11 text Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing Yaser Al-Onaizan editor Mohit Bansal editor Yun-Nung Chen editor Association for Computational Linguistics Miami, Florida, USA conference publication oh-schuler-2024-leading 10.18653/v1/2024.emnlp-main.202 https://aclanthology.org/2024.emnlp-main.202/ 2024-11 3464 3472