Qi Yang Huang
Author directory2026
What Signal Does Preemption Need? Benchmarking Preemption-Target Selection for SLO-Aware Reasoning LLM Serving
Qi Yang Huang | Alay Dilipbhai Shah | Pragaash Ponnusamy | Hoang Anh Duy Le | Thomas A. Joseph | Adam Seligman | Zhaozhuo Xu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Qi Yang Huang | Alay Dilipbhai Shah | Pragaash Ponnusamy | Hoang Anh Duy Le | Thomas A. Joseph | Adam Seligman | Zhaozhuo Xu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
When the key-value (KV) cache fills, a serving engine must choose which request to evict. We present PreemptBench, comparing seven preemption-target policies across seven reasoning-model families and two benchmarks. Our service-level objective (SLO) breach rate counts every accepted request, including late completions, post-acceptance failures, and unfinished work; this survivorship correction prevents deferred backlog from being omitted from the metric. In the low-breach regime, where last-in, first-out (LIFO) breach is below 15%, young-target-biased policies form a lower-breach cluster than policies without this bias, including random and largest-KV, consistent with a stable incumbent cohort; graded eviction depth is secondary. Within a pure remaining-decode rule, replacing the online predictor with a privileged empirical statistical length oracle, constructed from prior temperature-zero runs, yields no significant gain; neither significantly outperforms LIFO. It summarizes a prompt-conditioned length distribution, not the current execution’s realization, and is not SLO-optimal. Finally, the uncensored single-stream completion-time tail defines a conceptual feasibility floor, but our traces provide only a censored low-load proxy. Post-hoc rescoring across 600–1800-second deadlines finds aggregate memory-freeing reversals at tighter SLOs, but none at 1800 seconds. Thus LIFO is a strong conditional baseline, not a universal optimum.