Do leading LLM routers actually outperform simpler alternatives?

LLMRouterBench is introduced as "a large-scale benchmark and unified framework for LLM routing" comprising "over 400K instances from 21 datasets and 33 models," built to "systematically re-evaluate the field." Re-evaluating existing routing baselines under this shared framework, the paper reports that "many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline." Across the metrics the paper names — Gain@R, AvgAcc, Gain@B, and Gap@O — "leading routing methods (EmbedLLM, GraphRouter, MODEL-SAT, and Avengers) yield similar outcomes."

The paper's explanation centers on evaluation, not on routing itself: because prior routing papers used different model pools, datasets, and metrics, the field could not tell whether apparent gains came from a genuinely better router or a more favorable test setup. Put under one framework — a Collector, Evaluator, and Adaptor sharing "consistent train-test splits given the same random seed" — the differences narrow. The one standout, Avengers, "achieve such performance primarily through clustering and do not require neural-network training," which the paper reads as evidence that "lightweight routers, which are inexpensive to develop and straightforward to update, may be sufficient in practice."

This complicates how to read the cost and quality gains claimed by Can routers select the right model before generation happens? and Can cheaper language models match GPT-4 accuracy?: those figures were measured on separate benchmarks, and LLMRouterBench's point is that separate benchmarks cannot be compared head to head. It also qualifies Can routing beat building one better model?, whose clustering-based approach is the same family the paper credits with matching trained routers without training, and it gives empirical teeth to the taxonomy in Can five components unify all LLM routing approaches?, since the Collector/Evaluator/Adaptor framework is what makes those five components testable across methods rather than taken on any one paper's self-report.

The excerpt does not name which "simple baseline" the commercial routers failed to beat, which commercial routers were tested, or the absolute accuracy and cost numbers behind the comparable-performance claim — only that four named methods converge on four named metrics. It also does not establish that routing itself is not worthwhile: the paper still confirms "strong model complementarity" as "the central premise" of the field. The implication, at the strength the excerpt supports, is narrower: progress in routing research has outpaced the field's ability to tell routers apart, so claims of improvement from any single routing paper should be read against a shared benchmark rather than that paper's own reported baseline.