The generation quality of large language models (LLMs) is often improved by utilizing inference-time scaling methods such as Chain-of-Thought (sequential scaling) or Self-Consistency (parallel scaling). We introduce Roster of Experts (RoE), a training-free inference algorithm for Mixture-of-Experts (MoE) models that defines a third, orthogonal scaling direction. Rather than generating longer outputs or aggregating multiple sequences, RoE improves the next token prediction distribution by treating a single MoE as a dynamic ensemble of internal subnetworks. It injects controlled stochasticity into the expert routing mechanism via Gumbel-Top-k sampling, enabling the model to aggregate multiple diverse subnetworks into a single, higher-quality token prediction. Crucially, because RoE improves the model’s output distribution rather than sampling from it, it is the only approach in this family that is applicable to open-ended generation tasks where sequence-level aggregation is infeasible. Furthermore, RoE is orthogonal to existing scaling methods; combining RoE with Self-Consistency yields greater accuracy than either method alone. To overcome the computational cost of multiple forward passes at inference, we employ an efficient batching strategy that exploits hardware parallelism to minimize latency, requiring no fine-tuning of model parameters.

Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their substantial computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters in flash memory, but bringing them on demand to DRAM. Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this hardware-informed framework, we introduce two principal techniques. First, “windowing” strategically reduces data transfer by reusing previously activated neurons, and second, “row-column bundling”, tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.