@inproceedings{bie-etal-2026-osprey,
title = "Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding",
author = "Bie, Fengxiang and
Jian, Yuqing and
Yu, Yifan and
Zhou, Zhongzhu and
Shao, Zelei and
Athiwaratkun, Ben and
Song, Shuaiwen Leon and
Xu, Chenfeng and
Wu, Xiaoxia and
Zhang, Tianyi",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1641/",
pages = "37292--37312",
ISBN = "979-8-89176-459-0",
abstract = "Speculative decoding is a critical technique for accelerating LLM inference. In practice, however, the speedup is fragile: drafters are typically trained against a narrow data distribution for a single target model, and their acceptance rate collapses under workload shifts. This is a striking inversion of modern LLM development, where target models are valued precisely for the broad generalization they acquire through large-scale pretraining. We argue that the natural remedy{---}pretraining{---}has been hard to apply to drafters because existing recipes are deeply target-specific: the drafter consumes the target{'}s hidden states and is distilled on the target{'}s logits, so any pretraining effort must be redone for every new target. We introduce Osprey, which instead bootstraps drafters from off-the-shelf pretrained small language models, treating broad pretraining as a reusable, target-agnostic asset and reducing per-target work to a lightweight adaptation step. Realizing this requires overcoming two challenges: small LMs are far deeper than a latency-bound drafter can afford, and their pretrained computation must remain intact while the drafter learns to ingest target hidden states and emit tokens in the target{'}s vocabulary. Osprey addresses both by pruning to a shallow backbone, restoring its language-modeling capability with target-agnostic next-token pretraining, and then adapting it to each target through vocabulary alignment, zero-initialized QKV expansion, and target distillation. Empirically, a single pretrained Osprey backbone transfers across targets and improves mean acceptance length by 16.1{\%} for Qwen3-8B, 21.2{\%} for Llama-3.3-70B-Instruct, and 22.7{\%} for the 229B MiniMax-M2.5 (with 17.5{\%} higher tokens per second), with the largest gains on out-of-domain and multilingual data. Our code is available at https://github.com/LeanModels/Osprey."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bie-etal-2026-osprey">
<titleInfo>
<title>Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding</title>
</titleInfo>
<name type="personal">
<namePart type="given">Fengxiang</namePart>
<namePart type="family">Bie</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yuqing</namePart>
<namePart type="family">Jian</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yifan</namePart>
<namePart type="family">Yu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zhongzhu</namePart>
<namePart type="family">Zhou</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zelei</namePart>
<namePart type="family">Shao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ben</namePart>
<namePart type="family">Athiwaratkun</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Shuaiwen</namePart>
<namePart type="given">Leon</namePart>
<namePart type="family">Song</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chenfeng</namePart>
<namePart type="family">Xu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xiaoxia</namePart>
<namePart type="family">Wu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tianyi</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Speculative decoding is a critical technique for accelerating LLM inference. In practice, however, the speedup is fragile: drafters are typically trained against a narrow data distribution for a single target model, and their acceptance rate collapses under workload shifts. This is a striking inversion of modern LLM development, where target models are valued precisely for the broad generalization they acquire through large-scale pretraining. We argue that the natural remedy—pretraining—has been hard to apply to drafters because existing recipes are deeply target-specific: the drafter consumes the target’s hidden states and is distilled on the target’s logits, so any pretraining effort must be redone for every new target. We introduce Osprey, which instead bootstraps drafters from off-the-shelf pretrained small language models, treating broad pretraining as a reusable, target-agnostic asset and reducing per-target work to a lightweight adaptation step. Realizing this requires overcoming two challenges: small LMs are far deeper than a latency-bound drafter can afford, and their pretrained computation must remain intact while the drafter learns to ingest target hidden states and emit tokens in the target’s vocabulary. Osprey addresses both by pruning to a shallow backbone, restoring its language-modeling capability with target-agnostic next-token pretraining, and then adapting it to each target through vocabulary alignment, zero-initialized QKV expansion, and target distillation. Empirically, a single pretrained Osprey backbone transfers across targets and improves mean acceptance length by 16.1% for Qwen3-8B, 21.2% for Llama-3.3-70B-Instruct, and 22.7% for the 229B MiniMax-M2.5 (with 17.5% higher tokens per second), with the largest gains on out-of-domain and multilingual data. Our code is available at https://github.com/LeanModels/Osprey.</abstract>
<identifier type="citekey">bie-etal-2026-osprey</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.1641/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>37292</start>
<end>37312</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
%A Bie, Fengxiang
%A Jian, Yuqing
%A Yu, Yifan
%A Zhou, Zhongzhu
%A Shao, Zelei
%A Athiwaratkun, Ben
%A Song, Shuaiwen Leon
%A Xu, Chenfeng
%A Wu, Xiaoxia
%A Zhang, Tianyi
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F bie-etal-2026-osprey
%X Speculative decoding is a critical technique for accelerating LLM inference. In practice, however, the speedup is fragile: drafters are typically trained against a narrow data distribution for a single target model, and their acceptance rate collapses under workload shifts. This is a striking inversion of modern LLM development, where target models are valued precisely for the broad generalization they acquire through large-scale pretraining. We argue that the natural remedy—pretraining—has been hard to apply to drafters because existing recipes are deeply target-specific: the drafter consumes the target’s hidden states and is distilled on the target’s logits, so any pretraining effort must be redone for every new target. We introduce Osprey, which instead bootstraps drafters from off-the-shelf pretrained small language models, treating broad pretraining as a reusable, target-agnostic asset and reducing per-target work to a lightweight adaptation step. Realizing this requires overcoming two challenges: small LMs are far deeper than a latency-bound drafter can afford, and their pretrained computation must remain intact while the drafter learns to ingest target hidden states and emit tokens in the target’s vocabulary. Osprey addresses both by pruning to a shallow backbone, restoring its language-modeling capability with target-agnostic next-token pretraining, and then adapting it to each target through vocabulary alignment, zero-initialized QKV expansion, and target distillation. Empirically, a single pretrained Osprey backbone transfers across targets and improves mean acceptance length by 16.1% for Qwen3-8B, 21.2% for Llama-3.3-70B-Instruct, and 22.7% for the 229B MiniMax-M2.5 (with 17.5% higher tokens per second), with the largest gains on out-of-domain and multilingual data. Our code is available at https://github.com/LeanModels/Osprey.
%U https://aclanthology.org/2026.emnlp-main.1641/
%P 37292-37312
Markdown (Informal)
[Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding](https://aclanthology.org/2026.emnlp-main.1641/) (Bie et al., EMNLP 2026)
ACL
- Fengxiang Bie, Yuqing Jian, Yifan Yu, Zhongzhu Zhou, Zelei Shao, Ben Athiwaratkun, Shuaiwen Leon Song, Chenfeng Xu, Xiaoxia Wu, and Tianyi Zhang. 2026. Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 37292–37312, Budapest, Hungary. Association for Computational Linguistics.