LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing

Large language model (LLM) routing assigns each query to the most suitable model from an ensemble. We introduce LLMRouterBench, a large-scale benchmark and unified framework for LLM routing. It comprises over 400K instances from 21 datasets and 33 models. Moreover, it provides comprehensive metrics for both performance-oriented and performancecost trade-off routing, and integrates 10 representative routing baselines. Using LLMRouter- Bench, we systematically re-evaluate the field. While confirming strong model complementarity—the central premise of LLM routing—we find that many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. Meanwhile, a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures. We further show that backbone embedding models have limited impact, that larger ensembles exhibit diminishing returns compared to careful model curation, and that the benchmark also enables latency-aware analysis. All code and data are available at https://github.com/ynulihao/ LLMRouterBench.

Introduction. The rapid evolution of large language models (LLMs) has led to a proliferation of publicly available models. In this landscape, LLM routing has emerged as an important direction: rather than relying on a single model, routing methods operate over an ensemble of LLMs and dynamically assign each query to the model best suited to handle it. Since

Related work. Routing for Performance. This paradigm aims to enhance collective performance of an ensemble of LLMs by routing each query to the model that is most likely to produce a correct answer (Yue et al., 2025; Zhang et al., 2025a; Fein-Ashley et al., 2025; Wang et al., 2025a). LLM-Blender (Jiang et al., 2023) selects top candidate models through pairwise ranking and fuses their outputs. RouterDC (Chen et al., 2024b) applies dual contrastive learning to boost routing accuracy. EmbedLLM (Zhuang et al., 2024) leverages compact model and query embeddings. MODEL- SAT (Zhang et al., 2025b) constructs capabilityaligned embeddings to train a lightweight LLM as a router. Avengers (Zhang et al., 2025d) shows that a clustering-based routing, combined with aggregation methods, can empower a set of small LLMs to surpass proprietary models.

Routing for Performance-Cost Tradeoff. This paradigm routes queries to models to tradeoff between performance and cost (Song et al., 2025; Jin et al., 2025; Patel et al., 2025; Fernandez et al., 2025; Guo et al., 2025b), as more capable models are often associated with higher financial and computational costs. Earlier studies primarily focused on routing between only two models (typically a small one and a larger one). HybridLLM (Ding et al., 2024) targets the tradeoff based on predicted query difficulty. FrugalGPT (Chen et al., 2024a) adopts cascaded inference from small to large models. RouteLLM (Ong et al., 2024) trains the router from preference data. More recently, research has extended to larger model ensembles. Avengers- Pro (Zhang et al., 2025c) are both build upon clustering-based routing. GraphRouter (Feng et al., 2025a) employs a heterogeneous graph to represent task-query-LLM interactions, formulating model routing as an edge prediction problem.

Method. We construct the LLMRouterBench to systematically support two dominant paradigms in LLM routing research: routing for performance and routing for performance-cost tradeoff.

3.1 Model Pools In the performance-oriented setting, we benchmark routing methods’ ability to exploit complementary strengths among comparable-sized LLMs.1 In contrast, the performance-cost setting intentionally includes models with substantial variations in size, capability, and cost to reflect realistic performancecost tradeoffs. This yields two pools totaling 33 models:

3.2 Datasets 3.3 Modular Design To support flexible integration of diverse models, datasets, and routing algorithms, LLMRouter- Bench (see Fig. 2) adopts a modular design built around three key modules: Collector, Evaluator, and Adaptor. The Collector exposes a unified API to candidate LLMs, handling caching, retries, and cost tracking while consolidating all model outputs into the standardized format described above. The Evaluator implements rigorous, dataset-specific evaluation metrics (see Appendix Table 9 for details) to ensure fair comparisons across models. The Adaptor converts the standardized format into algorithm-specific inputs for each routing method, maintaining consistent train-test splits given the same random seed. This modular design allows LLMRouterBench to interface with publicly available routing implementations with minimal modification, facilitating unified evaluation and faithful alignment with their original implementations.

Discussion. 4.2.1 Performance-Oriented Setting No single model rules every domain; models exhibit complementary strengths. A central premise of performance-oriented routing is that different models excel in various domains. As shown in Fig. 3, we find clear evidence of this complementarity: mathematics benchmarks are often led by models such as Intern-S1-mini or Qwen3-8B, code benchmarks by Qwen-Coder or Fin-R1, and logical and affective benchmarks by other models. This confirms the central premise of this field.

Top routing methods are comparable, but can be free of neural network training. LLM routing is not new, resulting in a substantial body of literature. Nonetheless, our results indicate that, despite continued methodological innovation, many routing approaches achieve broadly comparable performance in practice. As shown in Fig. 4, across multiple metrics (Gain@R, AvgAcc, Gain@B, and Gap@O), leading routing methods (EmbedLLM, GraphRouter, MODEL-SAT, and Avengers) yield similar outcomes. Note that the Avengers achieve such performance primarily through clustering and do not require neural-network training. This observation has two implications. First, it is favorable from a deployment standpoint: lightweight routers, which are inexpensive to develop and straightforward to update, may be sufficient in practice. On the other hand, the lack of differentiation among leading methods suggests that a Routing remains far from its capability ceiling: current methods often miss the lone correct model. Although top routing methods perform similarly, which may raise the question of whether further gains are available, The GaptoOracle, shown in Fig. 4 (d), reveals a significant gap to the routing upper bound given by the Oracle baseline. A key contributor is model-recall failure: when only one or a few candidate models produce the correct answer, current routers often fail to select them. As analyzed in Fig. 5, this issue accounts for a non-trivial portion of the remaining error. For example, on queries where at most three experts an- Adding more models yields diminishing returns, but a well-chosen subset can make a difference. A common hypothesis in the field (Jiang et al., 2023) is that expanding the candidate pool should increase complementarity and therefore improve collective performance. However, using the Oracle baseline, we find clear diminishing returns as more models are added (Fig. 6). The largest gains occur when moving from a very small pool to a moderate one, after which additional models contribute only marginal improvements. By contrast, how we select a small subset matters substantially: comparing random selection to choosing the top-k models by average accuracy, the latter consistently delivers stronger performance. This suggests that careful curation can outweigh simply scaling the pool; a carefully selected moderate pool plus a robust router may offer most of the achievable benefit without the overhead of maintaining a very large pool. Future work should study model pool curation jointly with routing: selecting a small set that maximizes complementarity, potentially via coverage or diversity-style selection.

Models complement each other on performance and cost-efficiency. Similar to the performanceoriented setting, we observe complementarity among models in terms of performance and cost. As Appendix Table 5 shows, while GPT-5 achieves the best average accuracy and leads on HLE, substantially cheaper models such as Qwen3-235B and DeepSeek-R1 can match proprietary-level accuracy on selected mathematics and QA tasks. These results support the central premise behind routing for performance-cost tradeoffs.

Conclusion. We present LLMRouterBench, a large-scale benchmark and unified framework for LLM routing under both performance-oriented and performance-cost trade-off settings. By consolidating over 400K instances spanning 21 datasets and 33 models, and by introducing comprehensive metrics together with 10 representative baselines, LLMRouterBench provides a solid foundation for systematic study in this rapidly evolving area. Our results reaffirm the central motivation of LLM routing—strong complementarity across models—but also challenge several prevailing claims in the literature. Under a unified evaluation, most routing methods collapse to similar performance, and multiple recent approaches, including widely deployed commercial routers, fail to reliably outperform a simple baseline. A large and persistent gap to the Oracle remains, driven primarily by systematic model-recall failures rather than insufficient ensemble capacity. We further demonstrate that common design choices, such as the selection of backbone embedding models or aggressive scaling of ensemble size, yield limited gains in practice. Additionally, we show that our benchmark enables extending routing to tradeoffs for latency.

Limitations. Our work has three limitations. First, while we evaluate a broad set of routing methods, we do not cover all existing approaches. Given the large and growing number of routing methods, we focus on recent approaches with publicly available implementations. LLMRouterBench is modular by design, allowing additional routers to be integrated via lightweight adapters without reimplementing the full pipeline. Second, the benchmark is built from 21 widely used datasets that reflect common evaluation regimes for contemporary lightweight and flagship LLMs. Other settings, such as domain-specific verticals, very long-context tasks, and multimodal benchmarks, are not included. The routing formulations, metrics, and analysis procedures used in LLMRouterBench are generic and can be applied to these settings by adding new dataset evaluators. Third, the latency analysis is approximate. We estimate latency using token-level usage statistics together with throughput figures reported by Open- Router. These estimates correspond to a specific provider configuration and should be interpreted as indicative rather than definitive.