Jared Collette
Author directory2026
OpenDC-Infer-LC: Benchmarking Production-Valid Long-Context and RAG LLM Serving
Yue Yang | Maxwell Twelftree | Jared Collette | An-Chi He | Luke Borgnolo | David Lemphers
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Yue Yang | Maxwell Twelftree | Jared Collette | An-Chi He | Luke Borgnolo | David Lemphers
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Long-context and RAG workloads expose production bottlenecks that raw tokens/s and offline accuracy do not capture, including prefill cost, KV-cache pressure, prefix reuse, routing, and latency SLOs. We introduce OpenDC-Infer-LC, a hardware- and backend-neutral benchmark for evaluating complete data-center LLM serving stacks under long-context and retrieved-context workloads. The benchmark ranks systems by SLO-qualified, quality-preserving goodput: valid output-token throughput at the highest load satisfying p95 TTFT, p95 TPOT, p95 end-to-end latency, and success-rate targets. Because RAG and LongBench quality depends on model ability, retrieval difficulty, and task difficulty, OpenDC-Infer-LC uses a relative quality guardrail against a low-load reference run for the same model and workload, while reporting absolute quality separately. Pilot results on AMD MI355X and NVIDIA B200 show that deployable capacity depends strongly on backend scheduling, cache reuse, context length, routing, quantization, and runtime maturity.