Cheap-model-first systems save money, but checking and escalating takes time — how big is that hidden delay?

This explores the hidden time cost of query cascading: you try a small, cheap model first, check whether its answer is good enough, and pass the query to a larger model only if it isn't. To be direct, the corpus has no head-to-head benchmark that puts a number on cascade latency. What it does have is a set of neighboring designs that show where the delay comes from and how other systems avoid it. Together they explain the cost better than a single number would.

The main point comes from work on routing. A cascade decides *after* generation: it has to produce an answer and judge it before it knows whether to escalate. A router decides *before* generation, estimating how hard the query is and sending it straight to one model Can routers select the right model before generation happens?. RouteLLM and Hybrid-LLM get 40–50% cost reductions this way, about what cascades promise, while calling only one model per query. So cascade latency isn't a fixed overhead. It grows with how often you escalate. Every hard query pays for a wasted cheap generation, then the check, then the expensive generation. Easy queries pay almost nothing extra. Routing can go further than matching a single strong model: Avengers-Pro sends queries to specialized models by topic cluster and beats GPT-5-medium on accuracy, or matches it at 27% lower cost, without running a serial fallback chain Can routing beat building one better model?.

The second lesson is that the waiting comes from doing things in sequence, not from the number of model calls. Two notes from reasoning research make this concrete. GRAM explores several reasoning paths side by side instead of deepening one path step by step, which avoids the serial delay that comes with going deeper Can reasoning systems scale faster by exploring parallel paths instead?. And asynchronous verifiers run a checker *alongside* generation instead of after it, stepping in only when something goes wrong. On correct runs this adds almost no latency Can verifiers monitor reasoning without slowing generation down?. That is essentially a cascade's quality check moved off the critical path. It suggests the expensive part of cascading is not the check itself but making the user wait for it.

There's also a non-obvious interaction with caching. Prompt caching speeds up time-to-first-token by 13–31% in multi-turn agent tasks, but caching content that will never be reused can actually *add* latency Does prompt caching actually reduce agentic LLM costs?. Cascades break caching in a quiet way: when a query moves from the small model to the large one, the large model's cache has never seen that context, so the escalation step starts cold. In persistent agent setups where 82.9% of tokens were cache reads Do persistent agents really cost less per token?, losing that reuse on every escalation could cost more than the cascade's per-call savings.

In short, a cascade's latency penalty is roughly the escalation rate multiplied by (cheap-model generation time + check time + the cache warm-up you lose). The corpus doesn't measure that product directly. It does show that the two main alternatives, routing before generation and verifying in parallel, recover most of the cost savings while removing the serial wait. If you're choosing an architecture, the real question isn't how slow cascades are. It's whether your system can tell a query is hard before it answers, because if it can, you don't need the cascade at all.