Continual pre-training (CPT) adapts a pretrained language model to a target domain, but it often degrades general capabilities even when general-data replay is used to mitigate forgetting. We revisit this failure through replay supervision: prior CPT replay mainly tunes data mixtures and token budgets while applying one-hot next-token prediction (NTP) uniformly to domain and replay tokens. This target is natural for learning domain text, but on replay tokens it introduces target variance: it is a noisy, corpus-dependent proxy for preserving the frozen base model’s conditional behavior. We study self-distilled replay: domain tokens keep NTP, while replay tokens are trained to match the frozen base model distribution. Diagnostics show that this target substitution lowers target error, produces more directionally consistent replay gradients, and reduces base-behavior drift. Empirically, self-distilled replay improves the SmolLM3 domain–general trade-off over matched NTP replay across diverse token budgets and replay ratios. It also remains effective for Qwen3 models whose pretraining mixtures are unavailable: under the same mismatched, non-source replay setting, it improves both general score and PubMedQA in all eight matched comparisons. Overall, these results suggest supervising CPT replay to preserve base behavior, rather than fit empirical next tokens, as a way to reduce target variance.

Payment operations are a critical financial infrastructure, but the value of large language models in this domain remains unclear because payment rules change quickly, evidence is fragmented, and decisions depend on transaction state, participant role, region, and payment rail. Existing benchmarks do not isolate whether failures come from missing payment-rule knowledge, poor use of supplied evidence, or brittleness under imperfect harness inputs. We introduce BenchCompass, a payment-domain benchmark whose construction pipeline builds scenario-grounded tasks from typed evidence packs, applies LLM-based quality checks, creates task-input attack variants, and reserves final item admission for domain experts. The release contains an expert-reviewed Pro benchmark covering payment knowledge, context-grounded scenario reasoning, and Attacked Open robustness, plus a lower-assurance Normal pool for inspection and future curation. Across 16 model variants, BenchCompass shows qualitatively different failure modes: missing parametric payment knowledge, incomplete reasoning over supplied rules, and failure to reject plausible but invalid workflows. The benchmark remains unsaturated: the best frontier model reaches 89.6% on Open Context-Grounded Reasoning and 81.7% under attacked inputs, while a representative 32B open-weight model reaches 69.8% and 42.6%. The benchmark, data, and code are publicly available at https://github.com/ant-intl/BenchCompass.

Developing enterprise customer-service assistants with Reinforcement Learning (RL) critically depends on high-fidelity user simulators and well-aligned reward functions, since real-customer interactions are impractical for large-scale training and rule-based rewards fail to capture complex business requirements. We present CoEvoLoop, a framework that co-evolves the assistant policy together with these RL components through an expert-calibrated, multi-round loop. Beyond an LLM-based simulator for rollout generation and a multi-dimensional grader for rubric-based evaluation of service quality and compliance, CoEvoLoop integrates an expert annotation interface where sampled trajectories and grader judgments are reviewed and corrected. Each round, expert feedback updates the simulator and grader, after which the assistant is optimized with trajectory-level RL against the updated evaluation signal; monitoring interfaces support rollout inspection, dimension-wise failure attribution, and cross-round tracking. In an industrial evaluation, the system improves multi-turn service quality and reduces compliance violations over training rounds; component ablations and blind expert evaluation confirm that co-evolving the simulator and grader is load-bearing for these gains. We release the system demo code, along with a demo video.