Andrew Pulver
Author directory2026
PerQ: Efficient Evaluation of Multilingual Text Platform-Adaptation Quality
Dominik Macko | Andrew Pulver
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Dominik Macko | Andrew Pulver
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Since no metrics are available to evaluate specific aspects of a text, such as its personalization quality or quality of adaptation to a specific form, researchers often rely solely on large language models to evaluate such texts. Due to internal biases of individual language models, it is recommended to use multiple of them for combined evaluation, which directly increases costs of such evaluation. In this paper, a computationally efficient method (called PerQ) for evaluation of personalization quality (in terms of platform adaptation) of a given text (generated by a language model) is introduced, which is based on predicting the LLM judgments (i.e., a kind of metric distillation). A case study of comparison of generation capabilities of large and small language models shows the usability of the proposed metric in research, effectively reducing the waste of resources.