Yingying Liu
Author directory2026
OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
Weihuang Wen | Yingying Liu | Yichuan Liu | Wenqi Zeng | Li Zhou | Chumin Sun | Jie Sun | Tianshu Yu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Weihuang Wen | Yingying Liu | Yichuan Liu | Wenqi Zeng | Li Zhou | Chumin Sun | Jie Sun | Tianshu Yu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced frameworks (e.g., EAGLE-3) encode signals that are predictive of generation length. Building on this insight, we introduce **OUTLETS** (**Out**put-**Le**ng**t**h Prediction from **S**peculative Decoding Backbones), which repurposes the speculative backbone as a trajectory-aware length predictor. When its draft representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head and achieves lower MAE than the evaluated methods. Under saturated disaggregated serving, OUTLETS predictions enable standard scheduling policies to prioritize shorter requests and distribute requests more evenly across decoding instances, reducing short-request P99 latency by **34.8%**.