Andrew Pulver

Author directory

2026

Since no metrics are available to evaluate specific aspects of a text, such as its personalization quality or quality of adaptation to a specific form, researchers often rely solely on large language models to evaluate such texts. Due to internal biases of individual language models, it is recommended to use multiple of them for combined evaluation, which directly increases costs of such evaluation. In this paper, a computationally efficient method (called PerQ) for evaluation of personalization quality (in terms of platform adaptation) of a given text (generated by a language model) is introduced, which is based on predicting the LLM judgments (i.e., a kind of metric distillation). A case study of comparison of generation capabilities of large and small language models shows the usability of the proposed metric in research, effectively reducing the waste of resources.