Large language models (LLMs) trained with reinforcement objectives often achieve superficially correct answers via shortcut strategies, pairing correct outputs with spurious or unfaithful reasoning and degrading under small input perturbations. We introduce Coherence-Enhanced Policy Optimization (CE-PO), a model agnostic reward fusion framework that augments policy optimization with a differentiable Jacobian coherence signal along the generation pathway from prompt (X) to rationale (Y) to answer (Z). CE-PO estimates internal gradient influence through Jacobian-based sensitivities, uses a reshuffling-based residualization procedure to reduce shortcut-driven reward hacking, and fuses the resulting coherence score with task-accuracy via an L_p combiner that controls the strength of coherence regularization. The unified reward integrates with PPO/GRPO without architectural changes. Across multiple reasoning benchmarks, CE-PO improves accuracy over baselines by 5.49 on average (up to 9.58), while increasing the model’s faithful reasoning capability (12.14 % on average across five backbones).

As large language models (LLMs) are increasingly deployed in real-world applications, alignment is no longer governed by a single universal notion of safety or helpfulness, but instead by provider- or application-specific model specifications. These specifications are typically long, structured, and frequently updated, yet existing alignment pipelines lack a systematic mechanism to operationalize them as training signals. In this paper, we propose specification-grounded alignment, a new alignment paradigm that treats provider-authored model specifications as the primary alignment target rather than abstract principles or static benchmarks. To instantiate this paradigm, we introduce SpecAlign, a framework that synthesizes alignment data directly from specification documents. SpecAlign combines structured rule annotation, controllable specification instantiation, and multi-agent adversarial data synthesis to generate fine-grained, boundary-aware preference pairs that capture both compliant behaviors and meaningful specification violations. Experiments across multiple model specifications and backbone models demonstrate that training with SpecAlign consistently improves rule compliance while preserving general capabilities and avoiding over-conservative behavior. These results suggest that grounding alignment in explicit model specifications enables rapid, precise, and scalable adaptation of LLM behavior to evolving policy requirements.