FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

Abstract There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs—e.g. GPT-4, ChatGPT, J1-Jumbo—and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

Introduction. We are in the midst of an explosion of large language models (LLMs). The alluring possibilities of using LLMs for large-scale applications such as commerce, science, and finance have led a growing number of companies (OpenAI, AI21, CoHere, etc.) to offer LLMs as services. While LLMs such as GPT-4 achieves unprecedented performance in tasks such as question answering, using them for high-throughput applications can be very expensive. For example, ChatGPT is estimated to cost over $700,000 per day to operate [Cosa], and using GPT-4 to support customer service can cost a small business over $21,000 a month [Cosb]. In addition to the financial cost, using the largest LLMs encures substantial environmental and energy impact [BGMMS21, WRG+22], affecting the social welfare of current and future generations. There are many LLMs now available via APIs and they charge heterogeneous prices. The cost of using a LLM API typically consists of three components: 1) prompt cost (proportional to the length of the prompt), 2) generation cost (proportional to the generation length), and 3) sometimes a fixed cost per query. We compared the cost associated with using 12 different commercial LLMs from mainstream providers including OpenAI, AI21, CoHere and Textsynth (Table 1). Their cost can differ by up to 2 orders of magnitudes: for example, the prompt cost for 10M tokens is $30 for OpenAI’s GPT-4 but only $0.2 for GPT-J hosted by Textsyth. Given the heterogeneous cost and quality, how to effectively and efficiently leverage the full set of LLM options is a key challenge for pracitioners. If the tasks are relatively simple, then aggregating multiple responses from GPT-J [WK21] (whose size is 30x smaller than GPT-3) offers performance similar to GPT-3 [ANC+22], leading to financial and environmental savings. However, the performance of GPT-J can be much worse on difficult tasks [TLI+23]. Moreover, relying on one API provider is not reliable if that provider becomes unavailable, potentially due to spiking demand. Existing model ensemble paradigms such as model cascade [VJ04, WLM11] and FrugalML [CZZ20, CZZ22] were designed for predictive tasks with a known set of labels and do not account for the full capabilities of LLM. How to use LLMs affordably and accurately therefore calls for new approaches.

Our contributions. In this paper, we lay out our vision of a flexible framework that uses LLM APIs to process natural language queries within a budget constraint, termed FrugalGPT. As shown in Figure 1, we discuss three main strategies for cost reduction: prompt adaptation, LLM approximation, and LLM cascade. The prompt adaptation explores how to identify effective (often shorter) prompts to save cost. LLM approximation aims to create simpler and cheaper LLMs to match a powerful yet expensive LLM on specific tasks. LLM cascade focuses on how to adaptively choose which LLM APIs to use for different queries. To illustrate the potential of these ideas, we implement and evaluate a simple version of FrugalGPT using LLM cascade. On each dataset and task, FrugalGPT learns how to adaptively triage different queries in the dataset to different combinations of LLMs, including ChatGPT [Cha], GPT-3 [BMR+20] and GPT-4 [Ope23]. Our experiments show that FrugalGPT can save up to 98% of the inference cost of the best individual LLM API while matching its performance on the downstream task. On the other hand, FrugalGPT can improve the performance by up to 4% with the same cost. We believe this is only the tip of the iceberg and we hope FrugalGPT opens a new window toward reducing LLMs’ inference cost and improving its performances.

Related work. Related Works. Prompt Engineering. Prompt engineering has emerged as a discipline for crafting prompts to enhance LLMs’ performance across various applications. Recent developments include few-shot [BMR+20], chain-of-thought [WWS+22], knowledge enhancement [LLL+21, KSL+22], and numerous other prompting techniques [MDL+23, KTF+22, ZSH+22, DGSG22]. Existing prompt engineering approaches often aim to provide more detailed task explanations and in-context examples, resulting in longer and more expensive prompts. In contrast, this paper explores the use of concise prompts to reduce costs. Model Ensemble. Model ensembles, which involve combining multiple ML models for prediction, have gained popularity in supervised learning [VJ04, Fri02], unsupervised learning [YLLL14], semisupervised learning [GDMR22], and weakly supervised learning [DSP+17]. Model ensembles typically require white-box access to multiple models for training purposes, but LLM APIs are often black-box. Moreover, model ensembles necessitate querying all models for a single query, thereby increasing costs. System Optimization for LLMs. Numerous efforts have aimed to accelerate the training and inference time of modern deep learning models through system optimization [HMD15, CHSV17, Cas19, JZA19, RRWN11]. Recent work focuses on post-training quantization [BHS+22, YLW+23, XLS+22], training pipeline parallelism [LZG+21], and hardware-aware pruning [KFA23] tailored for LLMs. System optimization requires modifications to LLMs’ internal states (e.g., model weights), but many commercial LLM APIs do not release their models. Additionally, the rapidly increasing size of LLMs renders retraining highly expensive. ML-as-a-Service. LLM APIs constitute a crucial component of the rapidly expanding machinelearning-as-a-service (MLaaS) industry. Recent studies have demonstrated the diversity of different ML APIs’ predictions [BG18, KNL+20, CCZZ21] and proposed strategies for leveraging various classification ML APIs to improve performance [CZZ20, CZZ22]. The outputs of LLM APIs encompass the entire natural language space, but existing work requires a fixed (and known) label set. Moreover, both prompt choices and LLM API selections significantly impact generative tasks’ performance, resulting in a considerably larger optimization space than standard classification. The remaining part of the paper is organized as follows. We start by offering more context and the problem statement in Section 2. Next in Section 3, we present our visions on how to use LLM APIs affordability and accurately. Section 4 shows the empirical benefits of FrugalGPT using real-world LLM APIs (including GPT-3, ChatGPT, and GPT-4). Finally, we discuss future prospects in Section 5.

Method. 2 Scope and Problem Statement Natural language query answering. In this paper, we concentrate on the standard natural language query answering task, where the objective is to answer a query q sampled from a natural language query distribution Q. Various real-world natural language tasks, such as news classification, reading comprehension, and commonsense reasoning, can be formulated as query-answering problems.

LLM marketplace. We consider answering queries via the LLM market, which comprises K different LLM APIs, denoted by {fi(·)}K i=1. Each fi(·) : P 7→A is a function that, given a prompt p from the prompt space P, generates an answer from the answer distribution A. Note that to use LLM APIs, one has to convert each query q to some corresponding prompt first. LLM APIs are associated with their own cost, typically consisting of three components: a portion proportional to the length of the prompt, a portion proportional to the length of the generated answer, and (sometimes) a fixed cost per query. Formally, given a prompt p, the cost of using the ith LLM API is denoted by ci(p) ≜ ̃ci,2∥fi(p)∥+ ̃ci,1∥p∥+ ̃ci,0, where ̃ci,j, j = 0, 1, 2 are constants.

An illustrative example. Adapting the case study provided by [Cosa], assume a small business operates a customer service using GPT-4. The company caters to 15,000 customers each month, with each customer asking three questions twice a week, totaling 360,000 queries per month. Suppose for each question, its corresponding prompt averages 1800 tokens, and the answer is around 80 tokens. Considering that the input and response costs of GPT-4 are $0.03 and $0.06 per thousand tokens, respectively, the total monthly cost amounts to 360 × ($0.03 × 1800 + $0.06 × 80) ≈$21.2K. Such a high cost is prohibitive for many small businesses.

Problem statement: budget-aware LLM API usage. Our primary goal in this paper is leveraging LLM APIs within a budget constraint. Formally, this can be formulated as maximizing the overall task performance E(q,a)∈Q×A[r(a, ˆa(s, q))], while ensuring the average cost is bounded by a user-defined value b, i.e., E(q,a)∈Q×A[c(s, q)] ≤b. Here, a denotes the correct answer to the query q, ˆa(s, q) is the generated answer by some strategy s for query q, and c(s, q) is the associated cost for processing query q using strategy s. The reward function r(·, ·) measures how closely the generated answer aligns with the correct one. It is crucial to note that the search space for the strategy is vast, encompassing factors such as which prompts to use, which LLM APIs to employ, and how to aggregate their responses.

3 How to Use LLMs Affordably and Accurately Now we present our vision on how to use LLM APIs within a budget. As shown in Figure 1 (b), we discuss three cost-reduction strategies, namely, prompt adaptation, LLM approximation, and LLM cascade.

Strategy 1: Prompt adaptation. The cost of an LLM query increases linearly with the size of the prompt. Consequently, a logical approach to reduce the cost of using LLM APIs involves decreasing the prompt’s size, a process we refer to as prompt adaptation. Prompt selection (as illustrated in Figure 2 (a)) is a natural example of prompt adaptation: rather than employing a prompt containing numerous examples that demonstrate how to perform a task, one can retain a small subset of examples in the prompt. This results in a smaller prompt and subsequently lower cost. An intriguing challenge of prompt selection lies in determining which examples to maintain for various queries without compromising task performance. An additional instantiation is query concatenation (Figure 2 (b)). It is important to note that processing queries individually necessitates sending the same prompt to an LLM API multiple times. Therefore, the fundamental concept of query concatenation involves sending the prompt only once to the LLM API while allowing it to address multiple queries, thereby preventing redundant prompt processing. To accomplish this, several queries must be concatenated into a single query, and the prompt must explicitly request the LLM API to process multiple queries.

Discussion. Overall, FrugalGPT results in both accuracy gains and cost reduction. As illustrated in Figure 3(c), its cost is reduced by 80%, while the accuracy is even 1.5% higher.

LLM diversity. Why can multiple LLM APIs potentially produce better performance than the best individual LLM? In essence, this is due to generation diversity: even an inexpensive LLM can sometimes correctly answer queries on which a more expensive LLM fails. To measure this diversity, we use the maximum performance improvement, or MPI. The MPI of LLM A with respect to LLM B is the probability that LLM A generates the correct answer while LLM B provides incorrect ones. This metric essentially measures the maximum performance gains achievable by invoking LLM A in addition to LLM B. MPI between each pair of LLM APIs for all datasets is displayed in Figure 4. Overall, we observe significant potential within the LLM marketplace. For instance, GPT-C, GPT-J, and J1-L can all enhance GPT-4’s performance by up to 6% on the HEADLINES dataset. On the COQA dataset, there are 13% of data points where GPT-4 makes an error, but GPT-3 provides the correct answer. Although these improvement upper bounds may not always be attainable, they do demonstrate the possibility of utilizing more affordable services to achieve better performance.

Limitations. 5 Discussions, Limitations and Future Prospects The substantial cost of employing LLMs in real-world scenarios presents a considerable barrier to their widespread usage. In this paper, we outline and discuss practical strategies for reducing the inference cost of using LLM APIs. We also developed FrugalGPT to illustrate one of the cost-saving strategies, LLM cascade. Our empirical findings show that FrugalGPT can reduce costs by up to 98% while preserving the performance of cutting-edge LLMs. FrugalGPT lays the groundwork for optimizing task performance with LLM APIs under budget constraints; however, it has some limitations. To train the LLM cascade strategy in FrugalGPT, we need some labeled examples. And in order for the cascade to work well, the training examples should be from the same or similar distribution as the test examples. Moreover, learning the LLM cascade itself requires resources. We view this as an one-time upfront cost; this is beneficial when the final query dataset is larger than the data used to train the cascade. There are also other promising strategies for cost saving, such as speeding up attention computation itself, that we do not discuss here. Given the rapid development of LLM, this paper is not meant to be comprehensive or to provide a definitive solution.