Adam Seligman

Author directory

2026

When the key-value (KV) cache fills, a serving engine must choose which request to evict. We present PreemptBench, comparing seven preemption-target policies across seven reasoning-model families and two benchmarks. Our service-level objective (SLO) breach rate counts every accepted request, including late completions, post-acceptance failures, and unfinished work; this survivorship correction prevents deferred backlog from being omitted from the metric. In the low-breach regime, where last-in, first-out (LIFO) breach is below 15%, young-target-biased policies form a lower-breach cluster than policies without this bias, including random and largest-KV, consistent with a stable incumbent cohort; graded eviction depth is secondary. Within a pure remaining-decode rule, replacing the online predictor with a privileged empirical statistical length oracle, constructed from prior temperature-zero runs, yields no significant gain; neither significantly outperforms LIFO. It summarizes a prompt-conditioned length distribution, not the current execution’s realization, and is not SLO-optimal. Finally, the uncensored single-stream completion-time tail defines a conceptual feasibility floor, but our traces provide only a censored low-load proxy. Post-hoc rescoring across 600–1800-second deadlines finds aggregate memory-freeing reversals at tighter SLOs, but none at 1800 seconds. Thus LIFO is a strong conditional baseline, not a universal optimum.