Could picking the right AI model per question beat spending big to build one smarter model?
Can routing ever beat model improvement as a performance lever?
This explores whether picking the right model for each question can beat building a better single model, and when that choice is the better investment.
This explores whether choosing which model handles each question can do better than building one stronger model, and under what conditions. The short answer from the corpus is yes, sometimes by a lot. One system groups incoming questions by topic and sends each group to whichever model handles that kind of question best. It scored about 7% higher than GPT-5-medium, or matched it at 27% lower cost. An earlier version of the same idea used ten 7B models and beat GPT-4.1 and GPT-4.5 Can routing beat building one better model?. Simpler routers like RouteLLM don't aim for higher accuracy. They estimate how hard a question is before any answer is written and send easy questions to cheaper models, which cuts costs by 40-50% Can routers select the right model before generation happens?. The market already behaves this way. Open-weight models serve over 30% of tokens on OpenRouter even where closed models are more capable, because users trade capability against cost and speed How much do open-weight models actually get used in practice?.
The surprising part is where the remaining gains are. A benchmark of routing methods found that most methods end up at about the same performance, and all of them fall well short of a perfect router. The gap doesn't come from having too few models to choose from. It comes from failing to recognize which model already knows the answer What's really holding back LLM routing performance?. So routing's ceiling is high, and the bottleneck is the router's judgment, not the models it can pick from. There is also a catch. Routing only counts as a real lever if the models behave differently from each other and if two rephrasings of the same question go to the same model. Some routers score well on accuracy but send paraphrased questions to different models, so their gains are partly an illusion When does routing between models actually matter?.
The bigger pattern goes beyond routing. Across the corpus, organizing what a model gets to work with often matters more than making the model bigger. When a code repository was reorganized around what the software actually does at runtime, weaker planning models matched stronger ones at finding the right code, and they used fewer tokens Can explicit behavior maps help weaker planners compete with stronger models?. Reasoning models often give up on promising solutions too early. Penalizing them for switching lines of thought too quickly at generation time raises accuracy with no retraining Why do reasoning models abandon promising solution paths?. Recommender systems show the same thing: design choices that fit the problem beat deeper networks What architectural choices actually improve recommender system performance?. Routing is one example of a broader lesson. Selecting and structuring often beats adding raw capacity.
Routing and model improvement don't have to compete. A deployed router keeps a record of which questions were hard, which models succeeded, and what happened afterward. That record can be turned into labeled training data for fine-tuning the models, so the router also helps improve them Can a routing harness generate its own training data automatically?. Routing adds its own risk, though. The layer that decides which model handles a request sits below prompt-level safety checks. An attacker who can manipulate it could push requests to a weaker or less-guarded model Can attackers manipulate which model handles a request?.
Sources 10 notes
Avengers-Pro achieves 7% higher accuracy than GPT-5-medium by routing queries to optimal models per semantic cluster, or matches its performance at 27% lower cost. Ten 7B models with routing previously surpassed GPT-4.1 and 4.5, suggesting selection is a stronger lever than scaling.
RouteLLM and Hybrid-LLM both achieve 40-50% cost reduction by routing to a single model based on query difficulty prediction, not response evaluation. Single-model routing minimizes latency compared to ensemble or cascade alternatives.
Analysis of 100+ trillion tokens on OpenRouter found open models like DeepSeek and Qwen captured large usage shares even where closed models lead on capability. Users trade cost, latency, and other factors against capability, creating persistent market heterogeneity.
LLMRouterBench shows routing methods converge with each other but remain far from oracle performance, with the gap driven primarily by failures to identify which models can answer correctly—not by insufficient model pool size.
High task accuracy is compatible with meaningless routing if all models behave identically or if semantically-equivalent queries reach different models. Empirically, KNN routers achieve high accuracy but fail under paraphrase, while prompted routers trade accuracy for consistency.
Show all 10 sources
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
Research shows that architectural choices like removing hidden layers, enforcing constraints on self-similarity, and using appropriate likelihood functions deliver better results than deeper or more complex models. This suggests that problem-specific design decisions matter more than raw representational capacity.
A deployed routing system records execution trajectories, capability demand estimates, and outcome data that can be converted into labeled training examples for fine-tuning and distillation, turning the harness into both a serving component and a difficulty labeler.
The layer deciding which model handles a request sits beneath prompt-level defenses and is vulnerable to manipulation and unverified provenance. Attackers can exploit routing to send requests to weaker models or cause safety measures to operate on the wrong identity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
- RouteLLM: Learning to Route LLMs with Preference Data
- Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
- When is Routing Meaningful? Diversity and Robustness in Language Model Societies
- LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- Large Language Models Think Too Fast To Explore Effectively