How does routing decide between models before generation happens?
This explores how a router picks which model to use *before* any text is generated — predicting which model fits a query rather than running several and judging the outputs afterward.
This explores how a router picks which model to use *before* any text is generated — predicting which model fits a query rather than running several and judging the outputs afterward. The corpus draws a sharp line here: routing is a *pre-generation* decision, fundamentally different from a reward model that scores responses after the fact. Systems like RouteLLM and Hybrid-LLM estimate a query's difficulty up front and send it to a single model, cutting cost 40–50% while keeping latency low — because they commit to one model instead of running an ensemble or cascade and paying for all of them Can routers select the right model before generation happens?. The whole bet is that you can predict how hard a question is before answering it.
How does the router make that prediction? Two flavors show up. The difficulty-estimation approach asks 'is this query hard enough to need the expensive model?' The semantic approach skips difficulty entirely and asks 'what *kind* of query is this?' — Avengers-Pro clusters queries by meaning and routes each cluster to the model that does best on it, beating a frontier model by 7% on accuracy or matching it at 27% lower cost Can routing beat building one better model?. The striking claim there is that *which* model you pick can be a stronger lever than building a bigger one: ten small models with a good router previously edged out much larger frontier systems. Selection beats scaling.
Routing also gets more complicated than 'pick one model.' In multi-agent settings the pre-generation decision explodes into four entangled choices made at once — how the agents should collaborate, how many you need, what role each plays, and which LLM backs each one. MasRouter learns all four jointly through a cascaded controller and still comes out ahead on both accuracy and cost What decisions must multi-agent routing systems optimize simultaneously?. So 'routing' is less a single switch than a small upstream planning problem.
The most surprising corner is that routing doesn't only happen *between* models — it can happen *inside* one. Hybrid reasoning models route at the token level, deciding which tokens deserve expensive deliberation, and recover ~91% of the gains of full reasoning that way. This connects to a deeper finding: RL post-training seems to teach a model *when* to reason rather than *how*, meaning the reasoning ability already exists and the real skill being learned is a routing-like deployment decision Does RL post-training create reasoning or just deploy it?.
Worth knowing where this idea bends. The clean 'decide-then-generate' picture assumes you can judge a query before touching it — but a related thread argues a model's *partial* output reveals information the original query couldn't express, and that uncertainty mid-generation is a better trigger for fetching help than any up-front guess Can a model's partial response guide what to retrieve next? When should retrieval happen during model generation?. That's the live tension: pure pre-generation routing is cheap and fast, but some of the best signal about what a query needs only appears once generation is already underway.
Sources 6 notes
RouteLLM and Hybrid-LLM both achieve 40-50% cost reduction by routing to a single model based on query difficulty prediction, not response evaluation. Single-model routing minimizes latency compared to ensemble or cascade alternatives.
Avengers-Pro achieves 7% higher accuracy than GPT-5-medium by routing queries to optimal models per semantic cluster, or matches its performance at 27% lower cost. Ten 7B models with routing previously surpassed GPT-4.1 and 4.5, suggesting selection is a stronger lever than scaling.
MasRouter shows that routing in multi-agent systems must jointly optimize collaboration topology, agent count, role allocation, and per-agent LLM assignment through a cascaded controller. This unified approach surpasses single-model routing by 3.51% accuracy while cutting HumanEval costs by 49%.
Evidence shows base models already contain reasoning capability in latent form; RL training optimizes deployment timing rather than capability creation. Hybrid models recover 91% of performance gains by routing tokens only, and activation vectors for reasoning strategies pre-exist before any RL.
ITER-RETGEN shows that iteratively using generated responses as retrieval queries substantially improves performance on multi-hop reasoning and fact verification. Generation acts as both answer producer and information-need clarifier, surfacing implicit gaps that the original query missed.
Active retrieval triggered by low token probability improves both accuracy and efficiency compared to one-shot or continuous retrieval. FLARE demonstrates that models signal genuine knowledge gaps through low confidence, enabling dynamic budget allocation to actual information needs.