Subword tokenization is widely used in modern large language models for open-vocabulary coverage and token efficiency. We revisit word-level tokenization by comparing it with SentencePiece Unigram under matched byte fallback settings, allowing both tokenizers to represent out-of-vocabulary units without unknown tokens. Using language-appropriate word-like segmentation, we evaluate models pretrained from scratch across six model configurations in English and, using a common model configuration, across four languages (English, French, Chinese, and Japanese), with vocabulary sizes of 10k, 50k, and 100k. Although subword tokenization is generally more token-efficient, raw accuracy varies across model configurations, languages, and vocabulary sizes, with word-level tokenization remaining competitive across the evaluated settings. Word-level tokenization also tends to achieve higher character-length-normalized accuracy, although this complementary metric does not isolate the effects of tokenization on model scoring or establish a universal advantage in language modeling quality. These results show that word-level tokenization can remain competitive with subword tokenization when open-vocabulary coverage and language-appropriate segmentation are provided. Our code and tokenizers are publicly available.

Supervised fine-tuning (SFT) is a core stage in post-training general-purpose language models, but modern SFT training corpora are often large, heterogeneous mixtures, making data selection important under fixed token budgets. Simple ranking approaches can be informative yet insufficient: ranking alone does not determine how to allocate a fixed token budget across examples. We study the role-specific use of likelihood-derived signals for token-budgeted subset construction. An external instruction-tuned reference model computes response-side negative log-likelihood scores to rank training examples. We then combine this ranking with base model likelihood filtering to restrict the candidate region and response-side cosine filtering to reduce redundancy in the selected subset. In the OLMo fine-tuning setting studied here, ranking examples only by reference-model likelihood underperforms the random baseline, whereas the same ranking becomes more effective when combined with base model likelihood filtering and cosine-based redundancy control. Our results support viewing token-budgeted SFT data selection as a budget-allocation problem rather than only as an example-ranking problem.

Item categorization (IC) aims to classify product descriptions into leaf nodes in a categorical taxonomy, which is a key technology used in a wide range of applications. Along with the fact that most datasets often has a long-tailed distribution, classification performances on tail labels tend to be poor due to scarce supervision, causing many issues in real-life applications. To address IC task’s long-tail issue, K-positive contrastive loss (KCL) is proposed on image classification task and can be applied on the IC task when using text-based contrastive learning, e.g., SimCSE. However, one shortcoming of using KCL has been neglected in previous research: false negative (FN) instances may harm the KCL’s representation learning. To address the FN issue in the KCL, we proposed to re-weight the positive pairs in the KCL loss with a regularization that the sum of weights should be constrained to K+1 as close as possible. After controlling FN instances with the proposed method, IC performance has been further improved and is superior to other LT-addressing methods.

Short Answer Grading (SAG) is a task of scoring students’ answers in examinations. Most existing SAG systems predict scores based only on the answers, including the model used as base line in this paper, which gives the-state-of-the-art performance. But they ignore important evaluation criteria such as rubrics, which play a crucial role for evaluating answers in real-world situations. In this paper, we present a method to inject information from rubrics into SAG systems. We implement our approach on top of word-level attention mechanism to introduce the rubric information, in order to locate information in each answer that are highly related to the score. Our experimental results demonstrate that injecting rubric information effectively contributes to the performance improvement and that our proposed model outperforms the state-of-the-art SAG model on the widely used ASAP-SAS dataset under low-resource settings.