The rising costs of training frontier AI models

The costs of training frontier AI models have grown dramatically in recent years, but there is limited public data on the magnitude and growth of these expenses. This paper develops a detailed cost model to address this gap, estimating training costs using three approaches that account for hardware, energy, cloud rental, and staff expenses. The analysis reveals that the amortized cost to train the most compute-intensive models has grown precipitously at a rate of 2.4× per year since 2016 (90% CI: 2.0× to 2.9×). For key frontier models, such as GPT-4 and Gemini, the most significant expenses are AI accelerator chips and staff costs, each costing tens of millions of dollars. Other notable costs include server components (15-22%), cluster-level interconnect (9-13%), and energy consumption (2-6%). If the trend of growing development costs continues, the largest training runs will cost more than a billion dollars by 2027, meaning that only the most well-funded organizations will be able to finance frontier AI models.

Introduction. The large and growing cost of training state-of-the-art AI models has become an important issue in the field of AI [1]. Improving AI capabilities demand exponential increases in computing power, as evidenced by both economic analysis [2] and the discovery of empirical scaling laws, which show that model performance improves with more parameters and training data [3, 4]. Dario Amodei, CEO of the AI lab Anthropic, has stated that frontier AI developers are likely to spend close to a billion dollars on a single training run this year, and up to ten billion-dollar training runs in the next two years [5]. Given this trend, some innovations, particularly those requiring large-scale training, may become inaccessible to all but the most well-funded organizations.

Although it is widely known that training the largest ML models is expensive, until recently there were few concrete estimates of training costs in the public domain. In collaboration with Epoch AI, the 2024 AI Index presented one of the most comprehensive datasets to date, estimating the costs of training runs based on cloud rental prices [6]. We build on that work with a more in-depth account of hardware, energy and R&D staff costs for both training runs and experiments, as well as a more detailed analysis of how costs are increasing over time. To our knowledge, our study is the most thorough analysis of model development costs to date.

Our methods are built upon a comprehensive database of notable machine learning models [7], and informed by interviews with industry experts. We consider three complementary approaches to measuring the cost of frontier models. The first approach estimates the hardware capital expenses (CapEx) amortized over the final training run, along with the cost of hardware energy consumption. By considering AI accelerator chips, other server hardware, networking hardware, and energy separately, this approach can provide more accurate training costs. We find that the most expensive publicly-announced training runs to date are OpenAI’s GPT-4 at $40M and Google’s Gemini Ultra at $30M. Among frontier models, defined as models within the top 10 most compute-intensive models when they are released, we find that training has become 2.4× more expensive per year since 2016 (90% CI: 2.0× to 2.9×).

We then compare this approach to the cloud-price approach that was first presented in the AI Index [6]. Instead of estimating hourly compute costs in detail, the cloud-price approach simply uses historical rental rates from cloud platforms. The cloud-price approach shows a similar growth rate (2.5× per year with a 90% CI of 2.1× to 3.1×), but

Method. Our investigation draws upon the Notable AI Models database, which documents 796 notable models across the history of machine learning [7]. Key details captured for each model include training compute, dataset size, and parameter count. To focus on the largest-scale models, we initially filtered the database to models that had training compute estimates and that were published on or after 1 October 2015 (the start of the large-scale ML era according to [8]) and up to 31 December 2023. This resulted in 276 selected models. For these models, we recorded the training time, hardware type and quantity, and utilization rate sourced from each model’s original publication, where possible.

For our main results, we examined 41 models that were historically at the frontier of compute. Specifically, we filtered for models that were in the top 10 of training compute as of their release.1 Appendix B.1 provides further details on this selection procedure and a comparison to three alternative methods.

In addition to the data on machine learning models, we compiled a dataset of historical hardware prices, allowing us to estimate training costs. This price dataset contained cloud rental prices and hardware purchase prices for 24 different hardware models (e.g. NVIDIA A100) between 2015 and 2023. In total there were 142 entries, 52 of which were purchase prices and 90 of which were cloud rental prices.

To estimate the cost of hardware for a training run, we first calculated the cost to acquire the necessary accelerator chips (GPUs/TPUs), servers, and networking hardware. This involved looking up historical prices for GPUs, or estimating production costs for TPUs. Further details are provided in Appendix A.1.

Hardware normally remains available for future use after a training run finishes, but its value depreciates over time due to hardware progress. We amortized the cost of a training run based on this depreciation. Specifically, we depreciated the value of hardware at a rate of r = 0.14 orders of magnitude per year, based on the growth rate of ML GPU price-performance [9].2 To get the value of the hardware at the start of training, we used the following formula:

After finding the initial value of the hardware, the amortized cost of the training run is then the portion of value that is lost during the training run. However, the training time is often difficult to determine, due to a lack of public information. More often, we were able to estimate the number of chip-hours: the product of the training time and the number of chips. So we substituted chip-hours for the training time and the number of chips, using a linear approximation. This led to our final formula for amortized training cost:

Up until Section 3.5, our results only account for the chip-hours of the final training run. In Section 3.5, we scale up the chip-hours to account for all experiments towards developing an AI model. Although the amortized cost model involves several estimates and approximations, our results are robust to reasonable changes in the method (see Appendix A.3 for further methodological details and Appendix B.3 for the sensitivity analysis).

In addition to the capital costs of hardware, we also considered the cost of energy consumed by hardware during model training. We estimated this using the following formula: where TDP is thermal design power and PUE is power usage effectiveness, which accounts for the overhead of data center power distribution and cooling. We selected the energy cost rate by year, hardware TDP by the hardware type, average power to TDP ratio by the hardware manufacturer, and the data center PUE by the ML model developer. These were set based on hardware manufacturers’ literature. However, some parameters such as average power to TDP ratio could not be found in technical specifications and had to be estimated. For references and method details, see Appendix A.4.

Discussion. The rapid growth in AI training costs will have a major impact on the future of AI development. Our findings suggest that if the current trend of 2.4x per year growth continues, then the amortized cost of frontier training runs will exceed one billion dollars by 2027. Given the potential bias in our estimates’ absolute values, this may happen even sooner—as suggested by cloud-price costs, and news reporting on training costs [5]. If realized, this level of investment is likely to drive rapid advances in AI capabilities, given the track record of scaling up AI models.

However, only a handful of large companies and government institutions have the financial resources to operate at this frontier. This concentration of AI development could limit the range of perspectives and approaches considered, especially from academia and broader society. Both AI developers and policymakers must grapple with the rapid AI advances brought on by increasing investment, as well as the tradeoffs involved in the concentration of AI development. On one hand, having few key players at the frontier could make it easier for them to coordinate on responsible AI development. On the other hand, this raises concerns about a lack of public oversight for such a powerful technology.

Conclusion. In this paper we used three approaches to analyze the cost of training ML models at the frontier. The first two approaches—one based on hardware purchase prices and energy costs, the other based on cloud rental prices—indicate that the amortized cost of compute for these training runs has grown by around 2.4x per year (90% CI: 2.0x to 2.9x) since 2016. This shows the large role of investment in driving AI progress.

Breaking down the total amortized model development cost for selected frontier models (GPT-3, OPT-175B, GPT-4 and Gemini Ultra), we found that R&D staff are a major component, making up 29–49% of the total. This motivates further research on the scaling of R&D labor with computing power.

The rapid exponential growth of costs over eight years suggests that growth is unlikely to stall in the next few years. However, frontier AI labs appear to face non-trivial challenges to scaling further. One such challenge is securing enough power capacity for increasingly large computing clusters. Analyzing potential bottlenecks such as this is an important topic for future work.

The rapid increase in AI investment is likely to drive major advances in AI capabilities. Given that total model development costs at the frontier are already over $100 million, these advances may only be accessible to the largest companies and government institutions. The concentration of such a powerful technology among a few key players raises questions about responsible development and deployment. Both AI developers and policymakers must engage with these issues and consider the tradeoffs involved. The stakes are high—decisions made now about the governance and trajectory of AI could have profound consequences for society.

Limitations. 4.3 Limitations While our study provides valuable insights into the growth of AI training costs, there are important limitations. The analysis relies on publicly available information, which may lead to biases or gaps in the dataset. Cost estimation methods are subject to uncertainties due to factors such as hardware depreciation rates and pricing dynamics. Moreover, our methods neglect several costs that are potentially significant, including the data center infrastructure apart from the training cluster, and the acquisition of data for model training.

Our results may also have limited generality. The trends observed for the selected frontier models may not generalize to the broader AI landscape, or specific AI domains such as language modeling. Rapid innovation could also lead to large gains in hardware and software efficiency that are difficult to predict from historical data. Further research on all of these unknowns would help refine our insights, and inform evidence-based strategies to respond to growing financial barriers in ML.