Is a compute crunch coming?

Source: Epoch AI (Luke Emberson, Jaime Sevilla) · 2026-05-25

Much has been made about AI-driven capex in the past year. Hyperscalers have been clamoring to construct massive data centers, spending hundreds of billions in the process. The St. Louis Fed estimates that AI-related investment contributed about 1 percentage point — almost 40% of the total — to US real GDP growth in the first three quarters of 2025, exceeding the IT investment contribution at the height of the dot-com boom. Whether the current AI buildout constitutes a bubble depends largely on whether there will be sufficient demand for the computing infrastructure being built.

It’s tough to estimate future demand for tokens, as it depends heavily on hard-to-forecast trends in capabilities and diffusion. However, we have a much more concrete picture of the supply side. In this article, we do our best to answer how many tokens per second the world could produce with the chips we have today.

We compare these supply estimates to several (imperfect) proxies for token demand and its growth trend, including the growth of tokens served across Google platforms, and the intensity of token usage today at the largest tech companies, extrapolated to all software engineers worldwide. These figures suggest that demand at current prices could be between 200 million and 4 billion tokens per second, growing by roughly 10× per year — plausibly outpacing supply growth in the near future, if not already. However, these estimates are highly uncertain. For one, we don’t know the average size of the models behind current demand. Aggregate token figures also obscure an underlying trend in model efficiency, which both lowers the cost of producing tokens at a given quality, and introduces new demand as additional use cases become cost-effective.

If these trends continue, a compute crunch is likely near — particularly for the long-context workloads that drive agentic AI. This will drive up the price of access to frontier capabilities for those willing to pay, while everyday users shift to cheaper, smaller models. It may also mean that AI companies increase their focus on developing more efficient ways to serve models. Because of efficiency gains, these shifts won’t necessarily mean a regression in the capabilities accessible to everyday users — inference and training efficiency are already improving fast enough that the smaller, cheaper models of tomorrow will quickly match today’s frontier.

Accounting for these inefficiencies reduces our per-GB200 NVL72 throughput from 640,000 tok/s to 400,000 tok/s.15 Applying the same factors to GB300 systems and then scaling up by the total number of systems produces the following estimates:

Our final estimate suggests the world’s Blackwell GPUs could currently deliver a combined 500 million to 20 billion output tokens per second, or between 150,000 and 7 million tokens per month for each person on earth. To put that in perspective, Google, which is likely the most avid token producer today through Google Search summaries, recently claimed that it was serving 1.2 billion tok/s across all its platforms (very likely including both input and output tokens). If we assume a ratio of 8,000:1,000 input to output tokens (many of these requests are presumably short-input search results), they would be serving around 130 million output tokens per second. This suggests that even if we lavishly insist upon serving every Google request with an expensive, trillion-parameter model, there is plenty of inference capacity to serve all the needs of their users.

In fact, it would not only be enough to serve Google, but all tokens worldwide. Exponential View estimates the total tokens processed across all providers at 40 quadrillion tokens per quarter, i.e., around 5 billion tokens per second, four times Google’s traffic. There would still be enough compute capacity to serve all these tokens with a Kimi K2.6-like model, at least in our short- and medium-context settings.

And through a combination of infrastructure deployment and more efficient chips, inference capacity is growing over time. The two relevant trends to track are growth in compute capacity and in memory bandwidth, which are growing exponentially at 3.4×/year and 4.1×/year, respectively. In the long term, the slower-growing factor will determine the overall growth of inference capacity — in this case, compute. In the short term, growth can be closer to 4.1×/year while memory bandwidth remains the primary bottleneck; this will particularly affect long context requests like those in software engineering, since those are more deeply memory-bound. In practice, when we model the growth for short context (8,000:1,000 input-output) and long context (128,000:1,000) inference loads, we find the difference to be minimal; the inference capacity of the world at fixed model size and context length is well modeled as growing at 3.4×/year.

How does this compare to the growth of demand for tokens, at a fixed model size and price? Unfortunately, it’s difficult to make a crisp comparison, but the proxies that we have suggest that demand is growing much faster. For instance, both the quantity of tokens processed by Google in the last year, and by all providers according to Exponential View, have been growing by around 10×/year.

From another angle, we can look at token demand from today’s most intensive AI users: software engineers. Recent reports claim that some of Apple’s software engineers are permitted to use up to $300 in tokens per day, which works out to about 5 million output tokens per day with Claude Opus 4.7 API pricing, or 25 million output tokens per day with Kimi K2.6.16 Another point of comparison comes from Meta, whose 85,000 employees used 60 trillion tokens in one month across the organization. That figure included both input and output tokens; assuming a 25,000:1,000 input-to-output token ratio, that would be around 1 million output tokens per day and employee.

There were about 30 million software engineers worldwide as of 2025 (estimates range from 20 million to 50 million), and Stack Overflow’s 2025 survey on AI usage suggested that only around 47% of developers used AI on a daily basis, as of mid-2025. If all SWEs using AI daily were using it as intensely as Meta or Apple, they would demand somewhere between 10 and 350 trillion tokens per day in aggregate, i.e., between 200 million and 4 billion tokens per second. At the longest context sizes of 128,000:1,000, today’s Blackwell chips would struggle to serve all this potential demand for coding agents using models as large as Kimi K2.6. It also seems likely that both the number of developers using AI, and the intensity of their use will continue to grow rapidly.

All of the proxies above are imperfect. We don’t know the composition of those tokens, whether growth is dominated by small or large models, or the increase in context lengths. The prices of many providers have changed significantly over the period, affecting demand.17 And this analysis entirely ignores the very rapid pace of improvement in inference efficiency, which increases the demand for models of fixed scale by improving capabilities, and decreases it by displacing them with smaller models.

Regardless, these proxies suggest very fast growth in demand at fixed model size and price, likely faster than supply is expanding. And this is compounded by trends towards longer-context usage, especially due to coding and other agentic use cases.

If the demand for AI is outpacing the capacity to serve large models, the predictable consequence is that the price of tokens from large models will rise. This suggests a “compute crunch” is near, if not already here. Indeed, Anthropic has taken measures like reducing quotas during peak hours and incentivizing off-time usage in efforts to manage demand.

In a “crunch”, will everyday users be priced out of AI? Not necessarily. We’ve seen very fast growth in inference efficiency. If that continues, current use cases could be served by smaller, more efficiently served models — arguably, this is what has enabled the fast growth in tokens served we’ve seen so far. The largest models could be reserved for the most productive applications, such as coding, where access to the latest capabilities justifies a high per-token price.