Qingyu Xiao

Author directory

2026

Classical Chinese poetry generation and evaluation with Large Language Models (LLMs) face two intertwined challenges: existing generation benchmarks rely on simplistic keyword inputs devoid of literary context, and no ground-truth benchmark exists for objectively validating poetry evaluation methods. We present a comprehensive framework addressing both challenges. On the **generation** side, we construct **Honglou-Poem**, the first narrative-driven poetry generation benchmark derived from Dream of the Red Chamber, where each entry provides rich author personas and narrative backgrounds—moving far beyond traditional keyword-to-poem paradigms. A ”Digital Poetry Party” pipeline evaluates 12 state-of-the-art LLMs in this challenging setting. On the **evaluation** side, we introduce **Poetry-Tree**, a benchmark with axiomatically ordered quality levels created through systematic merit degradation, providing objective ground truth for evaluator validation. Leveraging Poetry-Tree as a training signal, we employ **GRPO** reinforcement learning with Spearman’s correlation as the reward to train a dedicated poetry evaluation model, **Poet-4B**. Our Poet-4B achieves a Spearman’s 𝜌 of **0.8253**, dramatically improving from the base model’s 0.2929 and closely approaching the best closed-source model GLM-4.6 (0.8764) with over 100× fewer parameters—a gap of only 0.051 in Spearman’s 𝜌. We release all benchmarks, trained models, and a leaderboard to facilitate future research.