Jared Collette

Author directory

2026

Long-context and RAG workloads expose production bottlenecks that raw tokens/s and offline accuracy do not capture, including prefill cost, KV-cache pressure, prefix reuse, routing, and latency SLOs. We introduce OpenDC-Infer-LC, a hardware- and backend-neutral benchmark for evaluating complete data-center LLM serving stacks under long-context and retrieved-context workloads. The benchmark ranks systems by SLO-qualified, quality-preserving goodput: valid output-token throughput at the highest load satisfying p95 TTFT, p95 TPOT, p95 end-to-end latency, and success-rate targets. Because RAG and LongBench quality depends on model ability, retrieval difficulty, and task difficulty, OpenDC-Infer-LC uses a relative quality guardrail against a low-load reference run for the same model and workload, while reporting absolute quality separately. Pilot results on AMD MI355X and NVIDIA B200 show that deployable capacity depends strongly on backend scheduling, cache reuse, context length, routing, quantization, and runtime maturity.