Compact open-source language models lag behind their larger counterparts in agentic tool-use reliability, yet standard remedies face fundamental obstacles: supervised fine-tuning suffers from exposure bias, while reinforcement learning is hampered by sparse credit assignment over long tool-interaction trajectories. We introduce Skill-CDPO, a progressive framework that first acquires tool-use skills at inference time through static tool analysis and dynamic strategy refinement, then distills the resulting error-correction signals into parameter updates via Critical Step DPO (CDPO). CDPO identifies the specific trajectory steps where model capability is the bottleneck—through rollout divergence between a local policy and an expert model—and constructs distributional preference pairs from all cross-group rollouts at those steps, weighted by both step-level criticality and pair-level score gaps. This provides dense, fine-grained supervision without requiring a process reward model. We evaluate Skill-CDPO on three medical agent benchmarks—PubMed Search (a new PubMed-based deep research benchmark we contribute), CureBench, and MedBrowseComp—using an 8B-parameter deep research model. Skill-CDPO substantially outperforms SFT and trajectory-level DPO baselines and achieves competitive or superior performance compared to GPT-5.2 on retrieval-intensive tasks. Our code and data are available at https://anonymous.4open.science/r/Skill-CDPO-BF51.

Large language models deployed as multi-step agents are increasingly trained via Direct Preference Optimization (DPO) on trajectory-level comparisons. Yet standard DPO weights every step uniformly, penalizing correct early actions in a losing trajectory as severely as the causal error itself. We propose Divergence-Weighted DPO(dwDPO), a drop-in replacement that detects the first divergence step between competing trajectories using implicit rewards already computed by DPO, then concentrates the loss around it via an asymmetric kernel with a closed-form scale factor that preserves the Bradley-Terry preference model. dwDPO recovers standard DPO as a special case and provably achieves higher gradient concentration at the critical step than both uniform and monotonic-discount baselines. Extensive experimental results across six deep-research benchmarks demonstrate the superiority of dwDPO over state-of-the-art open-source baselines of comparable scale at no additional inference cost.