Toshiki Kuramoto
Author directory2026
What Counts as Hallucination? A Scoping Review of Tasks, Symptoms, and Convergence Expectations in LLM Research
Toshiki Kuramoto | Ryohei Kamei | Jun Suzuki
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Toshiki Kuramoto | Ryohei Kamei | Jun Suzuki
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Hallucination in large language models (LLMs) and related generative models has become a central issue for reliability and practical deployment. Yet prior work does not consistently define what is treated as hallucination or the level of expectation under which model outputs are evaluated. This study conducts a scoping review of 244 papers published between 2018 and 2025, analyzing definitions of hallucination, output-side symptoms, and Convergence Expectation (CE). The results show that approximately 90% of the papers provide some form of operational definition, but the symptoms targeted by these operationalizations vary substantially by task. "Addition" appears across many tasks as a common symptom, whereas "Transformation" and "Deviation" strongly depend on task structure and input constraints. Analyses by CE band further show that, even within the same task, the symptoms problematized as hallucination vary according to the strictness of expected output convergence. These findings indicate that hallucination is not a fixed failure inherent in model outputs, but a phenomenon shaped by task, input constraints, evaluation criteria, and the expectations of humans or evaluators. This study reframes hallucination as expectation mismatch and argues that definitions, symptom scope, and expectation levels must be made explicit when comparing research findings and designing evaluations.
2025
Predicting Fine-tuned Performance on Larger Datasets Before Creating Them
Toshiki Kuramoto | Jun Suzuki
Proceedings of the 31st International Conference on Computational Linguistics: Industry Track
Toshiki Kuramoto | Jun Suzuki
Proceedings of the 31st International Conference on Computational Linguistics: Industry Track
This paper proposes a method to estimate the performance of pretrained models fine-tuned with a larger dataset from the result with a smaller dataset. Specifically, we demonstrate that when a pretrained model is fine-tuned, its classification performance increases at the same overall rate, regardless of the original dataset size, as the number of epochs increases. Subsequently, we verify that an approximate formula based on this trend can be used to predict the performance when the model is trained with ten times or more training data, even when the initial training dataset is limited. Our results show that this approach can help resource-limited companies develop machine-learning models.