A router can perfectly predict how hard each question is and still be making completely empty decisions.
Can a router predict query complexity well but still fail as governance?
This explores the gap between a router that accurately scores how hard a query is and a router that actually governs where work goes — i.e., whether complexity prediction is even the right measure of a routing system's job.
This explores the gap between predicting query complexity and governing traffic well — and the corpus is unusually direct that these are different things. The cleanest statement is that high routing accuracy can coexist with completely vacuous routing When does routing actually matter for multi-model systems?. If the models you're routing among are behaviorally redundant, a perfect complexity predictor changes nothing — every path leads to roughly the same answer, so the decision is real on paper and empty in practice. Complexity prediction is necessary but not sufficient; governance also requires that the choice have consequences.
The second failure the corpus names is stability. A router can nail complexity on a clean query and then flip its decision when the same query is paraphrased When does routing actually matter for multi-model systems?. That's a governance failure even when predictive accuracy looks high on a benchmark, because governance means consistent, defensible treatment of equivalent inputs — not a good average score. A system that routes 'reset my password' and 'how do I reset my password' to different tiers isn't governing; it's reacting to surface form.
Third, most routing framings in the collection optimize for a single scalar — cost or latency. RouteLLM and Hybrid-LLM cut cost 40-50% by predicting difficulty and sending queries to one model before generation Can routers select the right model before generation happens?. That's a good objective, but it's an efficiency objective, not a governance one. Governance is multi-constraint: who is allowed to serve this, under what budget, at what policy tier. The notes that treat routing as governance make those constraints first-class — coupling semantic matching to explicit policy and budget rather than to complexity alone Can semantic capability vectors replace manual agent routing?. A complexity-only router has no place to even express 'this query is simple but must go to the compliant model.'
There's also a subtler point about what 'complexity' should route on. StructRAG shows that the useful signal often isn't difficulty on a single axis but which kind of structure the task demands — a table, a graph, an algorithm, a plain chunk Can routing queries to task-matched structures improve RAG reasoning?. A router that collapses all of that into one complexity score can be well-calibrated on that score and still send queries to the wrong regime, because it optimized the wrong dimension. And the payoff for getting selection right is large — routing to specialized models can beat frontier models outright Can routing beat building one better model? — which is exactly why measuring the router by prediction accuracy alone undersells what a governing router is supposed to do.
The thread across all of these: complexity prediction is a per-query estimate, but governance is a property of the whole system's behavior — differentiated options, robustness to rephrasing, explicit policy and budget constraints, and routing on the dimension that actually matters. You can score the first perfectly and fail every part of the second.
Sources 5 notes
High accuracy coexists with vacuous routing when models are redundant or policies are unstable under paraphrase. Meaningful routing needs a differentiated society and robustness to surface-form variants, trading against raw performance.
RouteLLM and Hybrid-LLM both achieve 40-50% cost reduction by routing to a single model based on query difficulty prediction, not response evaluation. Single-model routing minimizes latency compared to ensemble or cascade alternatives.
Versioned Capability Vectors embedded in HNSW indices couple semantic matching with policy and budget constraints, making capability discovery a first-class operation that scales sub-linearly as agent heterogeneity increases.
StructRAG demonstrates that selecting knowledge structure type based on query demands—via DPO-trained router choosing among tables, graphs, algorithms, catalogues, and chunks—improves knowledge-intensive reasoning over standard retrieval. The approach grounds this in cognitive load and cognitive fit theory from cognitive science.
Avengers-Pro achieves 7% higher accuracy than GPT-5-medium by routing queries to optimal models per semantic cluster, or matches its performance at 27% lower cost. Ten 7B models with routing previously surpassed GPT-4.1 and 4.5, suggesting selection is a stronger lever than scaling.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- MasRouter: Learning to Route LLMs for Multi-Agent Systems
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
- When is Routing Meaningful? Diversity and Robustness in Language Model Societies
- RouteLLM: Learning to Route LLMs with Preference Data
- Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing
- Revisiting RAG Ensemble: A Theoretical and Mechanistic Analysis of Multi-RAG System Collaboration
- Towards a Science of Scaling Agent Systems
- StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization