Can cheaper language models match GPT-4 accuracy?

LLM API pricing is wildly heterogeneous: comparing 12 commercial LLMs from OpenAI, AI21, CoHere and Textsynth, the paper finds costs "can differ by up to 2 orders of magnitude" — the prompt cost for 10M tokens is "$30 for OpenAI's GPT-4 but only $0.2 for GPT-J hosted by Textsynth." At scale this is not abstract: ChatGPT is cited as costing "over $700,000 per day to operate," and a worked example has a small business running GPT-4 customer service at 360,000 queries a month, averaging 1,800 prompt tokens and 80 generation tokens per query, landing at "≈$21.2K" monthly. Against that backdrop, FrugalGPT — the paper's instantiation of an LLM cascade — "can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost." A separate reported run shows cost cut 80% with accuracy 1.5% higher.

The paper frames cost reduction as three distinct strategies: prompt adaptation (shrinking or reusing prompts — prompt selection keeps only a useful subset of in-context examples, and query concatenation batches several queries into one prompt to avoid repeating it), LLM approximation (building a cheaper model to match an expensive one on a specific task), and LLM cascade (adaptively choosing which LLM APIs to call per query, querying sequentially "until a reliable response is obtained" rather than deciding up front). The cascade's gains rest on a diversity argument the paper formalizes as MPI (maximum performance improvement): the probability that a cheaper LLM answers correctly where a pricier one fails. On the HEADLINES dataset, GPT-C, GPT-J and J1-L each improve on GPT-4 by up to 6% by this measure; on COQA, "13% of data points" have GPT-4 wrong but GPT-3 right. Cascading lets a query stop at a cheap model when confident and escalate only when it isn't, rather than paying the expensive model's price on every query regardless of whether it was needed.

This cascade is the same system cited, by name, inside Can routers select the right model before generation happens? as the contrasting case to single-model routing — that note notes cascades "multiply latency by the number of models queried" versus a router's one-shot decision, but doesn't carry the pricing evidence or the MPI mechanism behind why cascading pays off; this note supplies both. The escalate-only-when-needed logic also parallels the case for Can small language models handle most agent tasks?, though FrugalGPT escalates per query based on measured difficulty rather than defaulting to small models by task type. It is the same adaptive-allocation principle as Can we allocate inference compute based on prompt difficulty?, applied to which model answers rather than how much compute one model spends.

The paper is explicit that training the cascade "requires some labeled examples" drawn from a distribution similar to the queries it will later see, and treats this as a one-time upfront cost that only pays off "when the final query dataset is larger than the data used to train the cascade" — the 98%/4% figures describe a cascade tuned to a specific task and dataset, not a general-purpose policy that transfers across tasks untrained. The pricing snapshot itself (GPT-4 at $0.03/$0.06 per thousand tokens, the 2-order-of-magnitude API spread) is dated to this paper's measurement and says nothing about how those prices move afterward. What the paper does establish is a mechanism — exploit response diversity across API-priced models to buy accuracy or cost, not both at once, but one at the other's near-fixed price.