Can you get near GPT-4 quality by only calling it when a cheaper model's answer looks shaky?

This explores whether you can get big-model quality without paying big-model prices, either by making small models smarter or by mixing small and large models so the expensive one only does the hard part. The corpus says yes, often. But the savings rarely come from a small model simply standing in for a large one. They come from changing the question from "which model?" to "which model, for which query?"

The clearest result is routing. FrugalGPT sends each query to a cheap model first and moves up to an expensive one only when the cheap answer looks shaky. It matches GPT-4 accuracy at up to 98% lower cost, or gets 4% more accuracy for the same spend Can cheaper language models match GPT-4 accuracy?. This works because different models get different questions right, so a lot of everyday traffic never needed the biggest model. The same reasoning applies to AI agents. Most of an agent's work is repetitive, well-defined steps like formatting, extracting, and calling tools. Small models handle those at 10–30× lower cost, which makes "small by default, large when needed" the sensible design Can small language models handle most agent tasks?.

This setup has a hidden dependency: the system has to know when to escalate. That makes confidence estimation an economic question as well as a technical one. One approach, XConf, judges whether a model is likely right by looking up how it did on past problems where it felt similarly confident. It matches the reliability of sampling ten answers and checking whether they agree, at a tenth of the cost Can past performance predict when a model will be right?. Cheap, trustworthy confidence signals are what make routing between small and large models pay off.

Small models can also be made better outright. Training a small model on a large model's right *and wrong* answers to function-calling tasks (DPO on teacher-generated pairs) closes much of the gap. Seeing failures is what fixes the strict output-format errors that ordinary fine-tuning leaves behind Can small models match large models on function calling?. Design matters too. Below a billion parameters, deep-and-thin models beat wide-and-shallow ones by several accuracy points at the same size Does depth matter more than width for tiny language models?. You can also keep training a small model far past the usual stopping point. Quality was still improving at 10,000 training tokens per parameter, though more slowly than standard scaling laws predict Does model quality plateau when training on extreme token ratios?. This trade is attractive because you pay the extra training cost once and pay less on every query afterward. Some researchers go further and propose scaling dimensions that don't depend on parameter count at all, such as latent thought vectors Can latent thought vectors scale language models beyond parameters?.

The part you might not expect: some weaknesses don't go away with size, so in those areas a bigger model buys you nothing. LLMs still pattern-match instead of actually running step-by-step numerical methods at every scale tested Do large language models actually perform iterative optimization?. A 70B model still misreads deeply nested sentences Why do large language models fail at complex linguistic tasks?. So the real cost question isn't only whether small can match large. It's also where large stops being worth paying for. One limit: the corpus offers few head-to-head studies of total cost of ownership, meaning training plus serving plus engineering. Most of the evidence here is about inference cost per query.