Greed is All You Need: An Evaluation of Tokenizer Inference Methods

Greed is All You Need: An Evaluation of Tokenizer Inference Methods Omri Uzan author Craig W Schmidt author Chris Tanner author Yuval Pinter author 2024-08 text Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) Lun-Wei Ku editor Andre Martins editor Vivek Srikumar editor Association for Computational Linguistics Bangkok, Thailand conference publication uzan-etal-2024-greed 10.18653/v1/2024.acl-short.73 https://aclanthology.org/2024.acl-short.73/ 2024-08 813 822