Dynamic benchmarking is an essential research direction in the evaluation of large language models (LLMs) to avoid data contamination. Although existing works have proposed LLM-based methods for dynamic benchmarking, there still lacks a unified paradigm for performance comparison among different LLMs, which hinders researchers from model selection for specific domains and tasks. To address this challenge, we propose a comprehensive evaluation suite called DyBenchEval, including a novel seed dataset across 4 domains and 7 tasks as well as multi-aspect evaluation metrics. A cross data filling method is utilized for high-quality data augmentation with domain/task-level similarity, aiming to overcome the data scarcity problem. For automatic metrics, we consider coherence, correctness, and diversity of LLM-generated evaluation samples, simultaneously measuring the respective quality of generated questions / answers and the relationship between them. Extensive experimental results on DyBenchEval reveal deficiencies in 11 mainstream closed-source and open-source LLMs when dealing with dynamic benchmarking across different domains and tasks.