Reasoning-capable large language models (LLMs) achieve strong performance on complex tasks but often exhibit overthinking after distillation, generating unnecessarily long chain-of-thought (CoT) reasoning even for simple inputs and incurring high inference cost. However, naively shortening reasoning length can degrade reasoning accuracy, as concise reasoning may be insufficient for certain inputs and lacks explicit supervision. We propose Auto Long-Short Reasoning (AutoL2S), a distillation framework that empowers non-reasoning LLMs to think thoroughly but only when necessary. AutoL2S first learns a lightweight switching token with verified long-short CoTs to enable instance-wise long-short reasoning selection. Then it leverages long-short reasoning rollouts induced by switching tokens within a GRPO-style loss to improve reasoning efficiency while maintaining accuracy. Experiments demonstrate that AutoL2S effectively reduces reasoning length up to 71% with minimal accuracy loss, yielding markedly better trade-off in token length and inference time while preserving accuracy. The code is available at https://github.com/amandaluof/AutoL2S.

Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently *evict* tokens at every step of a continuous-inference stream. Existing methods all share the same template: a per-step direct-attention score followed by *deterministic top-K* selection, which converts a single below-cutoff step into an irreversible verdict and permanently erases any subtly important token that direct attention cannot single out from noise. To address this challenge, we propose **Nexus Sampling**, a training-free eviction method that pairs *Nexus scoring*, an iterative walk over direct attention that surfaces bridge tokens, with *weighted reservoir sampling*, which retains tokens with inclusion probability in place of deterministic top-K. Theoretically, we show that Nexus Sampling dominates deterministic top-K in long-run survival of subtly important tokens. Empirically, at 80% KV cache eviction, Nexus Sampling matches dense attention within ∼1 point on LongBench while outperforming top-K baselines on retrieval-heavy tasks, with up to 10× smaller per-sequence cache memory. Our code is available at <https://github.com/duongduc388222/NexusSampling>.

When the key-value (KV) cache fills, a serving engine must choose which request to evict. We present PreemptBench, comparing seven preemption-target policies across seven reasoning-model families and two benchmarks. Our service-level objective (SLO) breach rate counts every accepted request, including late completions, post-acceptance failures, and unfinished work; this survivorship correction prevents deferred backlog from being omitted from the metric. In the low-breach regime, where last-in, first-out (LIFO) breach is below 15%, young-target-biased policies form a lower-breach cluster than policies without this bias, including random and largest-KV, consistent with a stable incumbent cohort; graded eviction depth is secondary. Within a pure remaining-decode rule, replacing the online predictor with a privileged empirical statistical length oracle, constructed from prior temperature-zero runs, yields no significant gain; neither significantly outperforms LIFO. It summarizes a prompt-conditioned length distribution, not the current execution’s realization, and is not SLO-optimal. Finally, the uncensored single-stream completion-time tail defines a conceptual feasibility floor, but our traces provide only a censored low-load proxy. Post-hoc rescoring across 600–1800-second deadlines finds aggregate memory-freeing reversals at tighter SLOs, but none at 1800 seconds. Thus LIFO is a strong conditional baseline, not a universal optimum.

Large Language Models (LLMs) are increasingly adopted across real-world applications, yet traditional evaluations rely on expensive, domain-specific ground-truth labels that are often unavailable or infeasible. We introduce a ground-truth-free evaluation framework focused on reasoning consistency and instruction following, shifting the emphasis from correctness—which is elusive without labels—to transparent, coherent, evidence-based reasoning. Each model response must include a direct answer, a structured multi-step explanation, and supporting evidence, all assessed via semantic similarity and output adherence checks. We further propose TopK-ReRank, which refines rankings by constructing a consensus answer from the most reliable models, reducing ambiguity across diverse reasoning styles. Experiments show that our framework outperforms existing label-free methods, including majority voting, triplet ranking, and peer-review approaches, providing a more interpretable and efficient alternative for evaluating LLMs in the absence of ground-truth labels.

Large Reasoning Models (LRMs) are often bottlenecked by the high cost of output tokens. We show that a significant portion of these tokens are useless self-repetitions — what we call “word salad” — that exhaust the decoding budget without adding value. Interestingly, we observe that LRMs are self-aware when trapped in these loops: the hidden states of ‘‘ tokens trailing each reasoning chunk exhibit patterns that allow us to detect word salad behavior on-the-fly via a single linear classifier. Once detected, a simple chop appended by a straightforward regeneration prompt yields substantial length savings with minimal quality loss. Our work offers WordSaladChopper (WSC) — a lightweight, turnkey component for LRM that is minimally invasive to its reasoning trajectory. Given its low overhead, strong savings, and the lack of semantic value of word salad tokens, we believe it is not too far-fetched to argue that WSC — or a similar component — is a must-have for all LRM applications with user experience in mind.

Long context capability is a crucial competency for large language models (LLMs) as it mitigates the human struggle to digest long-form texts. This capability enables complex task-solving scenarios such as book summarization, code assistance, and many more tasks that are traditionally manpower-intensive. However, transformer-based LLMs face significant challenges with long context input due to the growing size of the KV cache and the intrinsic complexity of attending to extended inputs; where multiple schools of efficiency-driven approaches — such as KV cache quantization, token dropping, prompt compression, linear-time sequence models, and hybrid architectures — have been proposed to produce efficient yet long context-capable models. Despite these advancements, no existing work has comprehensively benchmarked these methods in a reasonably aligned environment. In this work, we fill this gap by providing a taxonomy of current methods and evaluating 10+ state-of-the-art approaches across seven categories of long context tasks. Our work reveals numerous previously unknown phenomena and offers insights — as well as a friendly workbench — for the future development of long context-capable LLMs. The source code is available at https://github.com/henryzhongsc/longctx_bench.