Can routing beat building one better model?
Does directing queries to specialized models via semantic clustering outperform investing in a single frontier model? This challenges whether model improvement or model selection drives performance gains.
Avengers-Pro demonstrates that routing queries to different models based on semantic clustering can exceed the performance of any individual model in the pool — including frontier models. The mechanism: embed incoming queries, cluster by semantic similarity, evaluate per-cluster model performance-efficiency scores, and route each query to the highest-scoring model for its cluster.
Three results establish the claim:
- Performance: +7% average accuracy over GPT-5-medium (the strongest individual model in the pool) across 6 benchmarks
- Efficiency at parity: matches GPT-5-medium accuracy at 27% lower cost
- Efficiency at near-parity: reaches ~90% of GPT-5-medium performance at 63% lower cost
The earlier Avengers work made an even more striking claim: ten models of ~7B parameters each, with routing, surpassed GPT-4.1 and 4.5 across 15 datasets. This suggests the performance gain from optimal model selection can be comparable to the gap between model generations.
The architecture is lightweight: three operations at inference time (embedding, nearest-cluster lookup, score aggregation). The heavy work — fitting the clustering model and estimating per-cluster performance statistics — happens offline on a calibration set (70% for fitting, 30% for evaluation). This makes the approach deployable as a thin routing layer atop any model API ecosystem.
Since Can we allocate inference compute based on prompt difficulty?, Avengers-Pro adds a complementary optimization axis. Compute-optimal scaling asks "how much inference budget per query?" Routing asks "which model per query?" These are independent — a routing layer could be composed with per-query compute allocation for a two-dimensional Pareto optimization. Since Can inference compute replace scaling up model size?, routing extends this: you don't need a bigger model OR more compute — you need the right model for this specific query type.
The implication challenges the frontier model race: rather than building one model that dominates on everything, assembling a diverse pool of specialized-ish models with good routing may be both cheaper and more effective. This aligns with the heterogeneous architecture thesis in Can small language models handle most agent tasks? — routing makes the heterogeneous approach practical.
Inquiring lines that read this note 68
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can LLM recommenders match or exceed collaborative filtering performance?- Why do naive baselines outperform trained models in entity-level CRS evaluation?
- Can this distillation pattern apply beyond e-commerce to other latency-constrained domains?
- Why does single-model routing beat ensemble and cascade approaches on latency?
- What makes query complexity a better routing signal than response quality?
- Can routing enable heterogeneous SLM-first architectures at scale?
- Should model routing decisions account for prompt-tier dependencies?
- Can model routing and compute allocation work together as independent optimizations?
- Can routing systems prevent expert models from failing outside their specialty?
- Can hierarchical vector routing reduce context overhead while maintaining tool coverage?
- How do routers decide when to escalate from small to large models?
- Can multiple small models outperform a single large model with good routing?
- How does semantic clustering help decide which model handles each query?
- Can compute allocation and model routing be combined for better results?
- Why might diverse smaller models with routing beat one giant model?
- What makes routing a better investment than training larger models?
- Can embedding-cluster routing outperform a single frontier model?
- How does routing decide between models before generation happens?
- Does model capability still matter once coordination infrastructure is optimized?
- Can semantic routing couple similarity matching with resource constraints?
- How do pre-training and distillation enable minimal routing signals to work?
- What makes mixture-of-experts routing learn token-level specialization effectively?
- Why does Branch-Train-Merge fail without learned routing between experts?
- Can routing policies remain meaningful over behaviorally homogeneous model pools?
- Does small heterogeneous model architecture outperform large homogeneous pools economically?
- Can a router predict query complexity well but still fail as governance?
- Why is latency budget a constraint for e-commerce rankers?
- How do different feed-weighting schemes construct distinct network topologies at population scale?
- How does nesting optimization levels improve on traditional network depth?
- How should topology routing adapt to different task types?
- Can bilevel autoresearch discover new search mechanisms for the inner research loop?
- Can bilevel autoresearch succeed when the inner and outer loops use different models?
- What capability risks emerge when models are optimized for single domains?
- Can review effort alone keep pace with frontier model degradation?
- Why do most frontier models terminate early on long-horizon benchmarks?
- What hidden costs emerge when you fine-tune models for a single domain?
- Do different domains require different types of model investment?
- Can smaller models actually perform well on specific downstream tasks?
- Can scaling data alone solve performance gaps on long-tail concepts?
- How does the Ladder of Scales approach reduce search costs across model sizes?
- Do small models show different parameter efficiency patterns than large models?
- Which architectural choices matter most when a model must fit one billion parameters?
- Why do frontier models remain cost-effective despite higher token prices in production?
- Why do frontier models corrupt more documents than weaker models during workflows?
- How does workflow scale change the failure modes of frontier models?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can we allocate inference compute based on prompt difficulty?
Does adjusting how much compute each prompt receives—rather than using a fixed budget—improve model performance? Could smarter allocation let smaller models compete with larger ones?
complementary axis: compute allocation + model selection = two-dimensional optimization
-
Can inference compute replace scaling up model size?
Explores whether smaller models given more thinking time during inference can match larger models. Matters because it reshapes deployment economics and compute allocation strategies.
routing extends substitution: right model > bigger model
-
Can small language models handle most agent tasks?
Explores whether smaller, cheaper models are actually sufficient for the repetitive, scoped work that dominates deployed agent systems, rather than relying on large models by default.
routing is the mechanism enabling heterogeneous architectures
-
Can routers select the right model before generation happens?
Explores whether LLMs can be matched to queries by estimating difficulty upfront, before any generation begins. This matters because routing could cut costs significantly while preserving response quality.
single-model routing as the base case this extends to multi-model pools
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
- RouteLLM: Learning to Route LLMs with Preference Data
- When is Routing Meaningful? Diversity and Robustness in Language Model Societies
- MasRouter: Learning to Route LLMs for Multi-Agent Systems
- RAG-R1 : Incentivize the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism
- Revisiting RAG Ensemble: A Theoretical and Mechanistic Analysis of Multi-RAG System Collaboration
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
Original note title
test-time model ensembling via embedding-cluster routing surpasses any individual frontier model — model selection is a stronger lever than model improvement