Jin Wu
Author directory2026
Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks
Xiangyu Wang | Jin Wu | Xiaoyu Li | Chanjin Zheng | Yifeng Zhou
Findings of the Association for Computational Linguistics: EMNLP 2026
Xiangyu Wang | Jin Wu | Xiaoyu Li | Chanjin Zheng | Yifeng Zhou
Findings of the Association for Computational Linguistics: EMNLP 2026
Automated evaluation of creativity tasks remains challenging for LLM-as-a-Judge, as LLM is susceptible to biases such as verbosity bias and leniency bias. Such limitations are particularly evident in Contextually-Grounded and Procedurally-Structured Tasks (CGPST), a complex multi-step creativity task where inter-step dependencies, highly subjectivity, and wide scoring ranges lead to more unstable and biased judgments. Existing approaches either rely on task-specific training or directly apply LLM-as-a-Judge, both of which struggle to ensure reliable evaluation under such complexity. To bridge these gaps, we propose **CreaEval**, an automated **crea**tivity **eval**uator for CGPST that decouples typical LLM-as-a-Judge into analysis and judging. Correspondingly, CreaEval involves two critical phases: **Memory-augmented Analysis**, a SoT-LLM converts multi-step responses into structured evaluation evidence, incorporating cross-step memory; and **Evidence-based Judging**, a Judge-LLM uses the extracted evidence for judging without accessing raw responses. Comprehensive experiments on CGPST show that CreaEval achieves an average human-LLM agreement of 0.64 (QWK) on CGPST, improving over the second-best baseline (supervised training) by 0.17. The code is available at https://anonymous.4open.science/r/CreaEval-0058/.
PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback
Wei Xia | Jin Wu | Haoran Shi | Xiangyu Wang | Chanjin Zheng
Findings of the Association for Computational Linguistics: ACL 2026
Wei Xia | Jin Wu | Haoran Shi | Xiangyu Wang | Chanjin Zheng
Findings of the Association for Computational Linguistics: ACL 2026
Effective Automated Essay Scoring (AES) are expected to support both reliable assessment and actionable instructional feedback. However, existing approaches often treat scoring and feedback as separate components: neural scoring models provide limited interpretability, while Large Language Model (LLM)-based feedback is typically insensitive to learners’ proficiency levels. To address this fragmentation, this work proposes PsyScore, a psychometrically-aware framework that integrates diagnostic assessment with instructional scaffolding through a shared latent ability representation. PsyScore comprises three key modules: a Trait-Adaptive Neural IRT Scorer that incorporates the Graded Partial Credit Model (GPCM) into a neural architecture, enabling the precise estimation of student ability while maintaining psychometric interpretability, a ZPD-Scaffolded Feedback Generator, which conditions multi-agent feedback strategies on the diagnosed ability parameter to adapt instructional focus across different proficiency levels, and a Multi-Perspective Feedback Evaluation Strategy that assesses feedback quality via pairwise preference judgments and student revision simulations. Experiments on the ASAP++ dataset demonstrate that PsyScore achieves competitive scoring performance while providing more pedagogically aligned feedback.