What's really holding back LLM routing performance?

Even where routing methods converge on each other, LLMRouterBench finds they remain far from optimal. "A substantial gap remains to the Oracle, driven primarily by persistent model-recall failures," the paper reports, with the Oracle defined as the routing upper bound "given by the Oracle baseline." The paper locates "a key contributor" to this gap in cases "when only one or a few candidate models produce the correct answer" and "current routers often fail to select them" — a failure mode it ties to "a non-trivial portion of the remaining error."

The paper separates two different limits on routing performance that are easy to conflate: how good the best available router is relative to other routers (where methods converge, per the companion finding), and how good any router is relative to the theoretical ceiling of always picking the one model that would have answered correctly. That gap to the ceiling persists even among the best current methods, because the router's job is to identify the correct expert from a crowd, and that gets harder when the right answer comes from only "at most three experts." The paper ties a related lever to the same ceiling: because "adding more models yields diminishing returns" once the pool grows past "a very small pool to a moderate one," and because "comparing random selection to choosing the top-k models by average accuracy, the latter consistently delivers stronger performance," the paper argues "careful curation can outweigh simply scaling the pool" — curation narrows what the router has to recall correctly, rather than asking it to recall correctly across an ever-larger crowd.

This adds a ceiling to the companion note, Do leading LLM routers actually outperform simpler alternatives?: routers can be indistinguishable from each other while all remaining far from the Oracle, so convergence among methods is not evidence that routing is close to solved. It also qualifies Can routing beat building one better model?: that note's claim that model selection beats model improvement holds relative to any single model, but this paper's Oracle gap shows selection itself still falls well short of its own ceiling.

The excerpt gives no absolute size for the Oracle gap, no number for what share of error model-recall failure accounts for beyond "non-trivial," and does not test whether curated pools close the recall gap or merely reduce the number of models a router has to search. The implication the evidence supports is modest: the binding constraint on routing quality looks less like needing more models or a cleverer router architecture, and more like a model-identification problem — knowing, from the query alone, which specific model in the pool would get it right.