Tree-structured speculative decoding accelerates large language model inference by drafting multiple candidate tokens in parallel and verifying them against a target model. Dynamic tree expansion strategies further improve efficiency by using draft model confidence to prune unlikely branches and stop expansion early. However, we find that existing draft models exhibit severe overconfidence: they assign high probabilities to tokens that will be rejected, causing unreliable early stopping and wasted computation on incorrect branches.

We propose Calibrated Tree-Structured Speculative Decoding (CATS-SD), a training framework for tree-based confidence calibration. Our approach calibrates draft model confidence at two complementary levels: (1) token-level calibration that aligns draft-target token ranking, and (2) path-level calibration that supervises confidence over tree-structured decoding paths. We also introduce a calibration metric using Pearson correlation to quantify confidence-acceptance alignment. On Qwen2.5-3B-Instruct, CATS-SD achieves up to 11.3% higher acceptance length and 4.0% better confidence-acceptance correlation compared to the baseline, with end-to-end speedup improving from 2.51× to 2.71× on GSM8K. We validate these results across mathematical reasoning (GSM8K), multi-turn conversation (MT-bench), and code generation (HumanEval).

The effectiveness of large language models (LLMs) in conversational AI is hindered by their reliance on single-turn supervised fine-tuning (SFT) data, which limits contextual coherence in multi-turn dialogues. Existing methods for generating multi-turn dialogue data struggle to ensure both diversity and quality in instructions. To address this, we propose Review-Instruct, a novel framework that synthesizes multi-turn conversations through an iterative “Ask-Respond-Review” process involving three agent roles: a Candidate, multiple Reviewers, and a Chairman. The framework iteratively refines instructions by incorporating Reviewer feedback, enhancing dialogue diversity and difficulty. We construct a multi-turn dataset using the Alpaca dataset and fine-tune the LLaMA2-13B model. Evaluations on MT-Bench, MMLU-Pro, and Auto-Arena demonstrate significant improvements, achieving absolute gains of 2.9% on MMLU-Pro and 2% on MT-Bench compared to prior state-of-the-art models based on LLaMA2-13B. Ablation studies confirm the critical role of the Review stage and the use of multiple Reviewers in boosting instruction diversity and difficulty. Our work highlights the potential of review-driven, multi-agent frameworks for generating high-quality conversational data at scale.