Linfeng Shi
Author directory2026
The Evaluation Scaling Law: Toward More Reliable LLM Assessment via Multi-Prompt Diversity
Hongzhi Luan | Zhaoxin Huan | Linfeng Shi | Longfei Zheng | Xiaolu Zhang | Jun Zhou
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Hongzhi Luan | Zhaoxin Huan | Linfeng Shi | Longfei Zheng | Xiaolu Zhang | Jun Zhou
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large Language Model (LLM) evaluation is frequently challenged by prompt sensitivity, which limits the reliability of benchmarks relying on a single template. To address this, we systematically investigate a multi-prompt evaluation framework that aggregates performance across diverse templates, leveraging a Quality-Diversity (QD) prompt selection mechanism to ensure both variety and fidelity. Through extensive empirical analysis, we uncover a novel Evaluation Scaling Law: evaluation reliability scales consistently with template diversity. Specifically, our experiments demonstrate that evaluation variance diminishes as the scale of prompts increases, while ranking monotonicity exhibits a log-linear improvement relative to the number of templates. By decoupling a model’s intrinsic capability from template-specific fluctuations, our framework facilitates more reliable assessments of LLMs, thereby providing a solid foundation for benchmarking that enables more effective monitoring of training dynamics and rigorous comparisons among state-of-the-art models.