Yifeng Zhou
Author directory2026
Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks
Xiangyu Wang | Jin Wu | Xiaoyu Li | Chanjin Zheng | Yifeng Zhou
Findings of the Association for Computational Linguistics: EMNLP 2026
Xiangyu Wang | Jin Wu | Xiaoyu Li | Chanjin Zheng | Yifeng Zhou
Findings of the Association for Computational Linguistics: EMNLP 2026
Automated evaluation of creativity tasks remains challenging for LLM-as-a-Judge, as LLM is susceptible to biases such as verbosity bias and leniency bias. Such limitations are particularly evident in Contextually-Grounded and Procedurally-Structured Tasks (CGPST), a complex multi-step creativity task where inter-step dependencies, highly subjectivity, and wide scoring ranges lead to more unstable and biased judgments. Existing approaches either rely on task-specific training or directly apply LLM-as-a-Judge, both of which struggle to ensure reliable evaluation under such complexity. To bridge these gaps, we propose **CreaEval**, an automated **crea**tivity **eval**uator for CGPST that decouples typical LLM-as-a-Judge into analysis and judging. Correspondingly, CreaEval involves two critical phases: **Memory-augmented Analysis**, a SoT-LLM converts multi-step responses into structured evaluation evidence, incorporating cross-step memory; and **Evidence-based Judging**, a Judge-LLM uses the extracted evidence for judging without accessing raw responses. Comprehensive experiments on CGPST show that CreaEval achieves an average human-LLM agreement of 0.64 (QWK) on CGPST, improving over the second-best baseline (supervised training) by 0.17. The code is available at https://anonymous.4open.science/r/CreaEval-0058/.