Large language Models (LLM) routing reduces inference cost by selecting the most cost-effective model per query, but production deployment requires a router that adapts to an evolving model pool while adding negligible per-query overhead. Unsupervised retrieval-based routers satisfy both, but their support corpus must be built online from streaming traffic, and labeling each query requires running every candidate plus an LLM judge: prohibitively expensive at scale. We frame online corpus construction as stream-based active learning, exploiting a structural property of routing benchmarks: one or two models are globally near-optimal on the majority of queries, so the labeling budget should concentrate on the rare exception regions where selecting a specialist matters. We formalize this with a performance sparsity score, defined as the Kullback-Leibler divergence of a query’s per-candidate performance vector from uniform: high when performance concentrates on a few models and low when all candidates are equivalent. Since this score depends on performances unavailable at inference time, we predict it online from query embeddings with a lightweight regressor. Across four benchmarks, our method SparseNN approaches full-annotation routing performance with up to 44% fewer annotations in the stationary setting and 73% under topic shift. The results transfer to a fully held-out benchmark. A partial model annotation variant achieves higher cost savings, especially on large pools. Finally, both the corpus and the online estimator prove robust to model-pool changes.

We study model routing for Large Language Model (LLM)-based systems. A model, called the router, dynamically chooses which LLM should handle a given input/query. We challenge the assumption that complex routers are necessary for generalising to new candidate LLMs. We introduce ContextualRouter, a simple meta-evaluation framework that predicts per-model performance for new queries by retrieving similar past queries and reweighting model scores with lightweight attention. During inference, the router balances estimated performance and cost by adjusting a tunable cost penalty parameter. This allows the router to adapt dynamically to the addition or removal of LLMs without the need for retraining. Across five routing benchmarks (SPROUT, RouterBench, LiveBench, BigGenBench, and EmbedLLM), ContextualRouter matches the quality–cost trade-offs of other generalisable routers. Surprisingly, a simpler non-parametric baseline, k-nearest-neighbour averaging, performs comparably or better, achieving strong performance estimation, high NDCG, and substantial cost savings. Retrieval-based routers remain robust to k, embedding size, data sparsity, retrieval degradation, and generalise to unseen queries and models with as little as 1% historical data. These results suggest that effective retrieval alone enables generalisable LLM routing.