Controllable creative writing is a text generation task that requires both high generation quality and diverse outputs. However, existing methods often improve one aspect at the expense of the other. To address this challenge, we propose SERGE, a reinforcement learning (RL) framework that introduces a diversity-guided self-refinement mechanism, where the policy itself refines low-diversity candidate texts, and employs policy shaping to amplify the gradient contributions of samples with both high generation quality and diversity. We also propose a fine-grained reward modeling framework to provide reliable quality reward signals to support RL optimization. Experiments on the short-form video copywriting generation task show that SERGE outperforms the base model and conventional diversity-enhancement strategies on both diversity metrics (DisNGram, PairBLEU, CosSim) and generation quality, effectively alleviating the "diversity collapse” issue in RL. Our code is available at: https://github.com/peng-mn/SERGE.