Linfeng Shi

Author directory

2026

Large Language Model (LLM) evaluation is frequently challenged by prompt sensitivity, which limits the reliability of benchmarks relying on a single template. To address this, we systematically investigate a multi-prompt evaluation framework that aggregates performance across diverse templates, leveraging a Quality-Diversity (QD) prompt selection mechanism to ensure both variety and fidelity. Through extensive empirical analysis, we uncover a novel Evaluation Scaling Law: evaluation reliability scales consistently with template diversity. Specifically, our experiments demonstrate that evaluation variance diminishes as the scale of prompts increases, while ranking monotonicity exhibits a log-linear improvement relative to the number of templates. By decoupling a model’s intrinsic capability from template-specific fluctuations, our framework facilitates more reliable assessments of LLMs, thereby providing a solid foundation for benchmarking that enables more effective monitoring of training dynamics and rigorous comparisons among state-of-the-art models.