Multi-chapter novel generation requires models to balance two quality dimensions simultaneously: chapter-level reader engagement and global narrative rhythm. Existing approaches have mainly focused on planning, memory, and execution, while paying limited attention to the mismatch between training signals and reader experience. We propose **ChapterRL**, which reformulates multi-chapter generation as a reward design problem. Specifically, ChapterRL introduces two complementary reward signals. **Reading Depth Residual** (RDR) derives debiased chapter-level quality signals from reader continuation and completion behavior through genre–position stratified normalization. **Dynamic Rhythm Trajectory** (DRT) uses an arousal estimator to impose a threshold-gated penalty on systematic rhythm collapse in the later part of a story. The two signals correspond to chapter attractiveness and global narrative momentum, and are jointly integrated into reinforcement learning training. We evaluate the full ChapterRL policy on three automatic benchmarks and separately validate its RDR reward model on task-aligned narrative domains of the Chinese WritingPreferenceBench. Results show that ChapterRL improves chapter quality and rhythm stability over supervised fine-tuning and single-reward variants, while RDR exhibits strong narrative preference alignment as an efficient scalar reward model.Code is available at [ChapterRL](https://github.com/SorBetQ/ChapterRL).

Reinforcement Learning with Verifiable Rewards (RLVR) has shown significant promise for enhancing the reasoning capabilities of large language models (LLMs). However, prevailing algorithms like GRPO broadcast a uniform advantage signal across all tokens in a sequence. This coarse-grained approach overlooks the pivotal role of uncertain, high-stakes decisions during reasoning, leading to inefficient exploration and the well-documented problem of entropy collapse. To address this, we introduce UnCertainty-aware Advantage Shaping (UCAS), a model-free method that refines credit assignment by leveraging the model’s internal uncertainty signals. UCAS operates in two stages: it first modulates the response-level advantage using the model’s overall self-confidence, and then applies a token-level penalty based on raw logit certainty. This dual mechanism encourages exploration of high-uncertainty paths that yield correct answers while penalizing overconfident yet erroneous reasoning, effectively balancing the exploration-exploitation trade-off. Extensive experiments on five mathematical reasoning benchmarks show that UCAS significantly outperforms strong RLVR baselines across multiple model scales, including 1.5B and 7B. Our analysis confirms that UCAS not only achieves higher rewards but also promotes greater reasoning diversity and successfully mitigates entropy collapse.