Cloud AI providers mostly pay for servers, not electricity — so what markup do they actually tack on top of hardware costs?

This explores the gap between what AI infrastructure costs to run and what providers charge for it. To be direct: none of these notes measures a cloud provider's margin. They do describe both ends of the gap, and that tells you a lot about how big it is and why it moves.

Start with the cost side. Epoch AI's model of a one-gigawatt AI data center puts its annualized cost at about $8.5B. Servers account for roughly 60% of that, while energy is only about $0.6B a year What actually drives AI data center costs?. So the main thing a provider has to recover is the price of the hardware, not the electricity bill. The same pattern shows up inside AI labs: compute takes 54–62% of operating spending at Anthropic, Minimax and Z.ai, while staff costs stay under 25% Do compute costs really dominate AI company budgets?. Whatever margin exists sits on top of a cost base made mostly of hardware.

The clearest indirect signal of markup comes from self-hosting. A 54-scenario cost model found that running a small LLM on your own hardware pays for itself compared with commercial APIs in as little as 0.3–3 months. For large models it takes 3.5–69 months When does self hosting an LLM beat commercial API costs?. If buying the hardware outright can beat API prices that fast, then for high-volume work those prices sit well above the raw cost of compute. That's an inference rather than a measured margin, but it's a strong one.

The most surprising part is that the margin isn't fixed. It depends on how new a capability is. Epoch found that the price of a given level of benchmark performance drops 66% per quarter right after it first becomes state of the art. Within two years that slows to 32% per quarter. Their explanation is that whoever ships a new capability first briefly holds something like a monopoly, and competitors then chip away at it Do AI performance prices fall faster when capabilities debut?. Denain sees the same trend from the cost side: the cost of reaching a fixed capability falls roughly 5–10x a year Does inference cost remain a persistent economic burden?. Read together, these suggest providers earn their widest margins on whatever is newest, and those margins shrink fast as capabilities become common. That's also why benchmarks should report token counts rather than one-time dollar costs: the price per token keeps changing, while the number of tokens a task uses doesn't Should benchmarks report token counts instead of frozen costs?.

For a sense of how margins work elsewhere in the AI supply chain, look at training data. Mercor reported $2B in gross annualized revenue, but it pays 60–70% of that to contractors, leaving about $600–800M in net revenue Can AI data companies sustain margins beyond labor payout ratios?. That gap between headline revenue and what a company actually keeps likely applies to compute resellers too. If you want an actual percentage for AWS, Azure or GCP on AI workloads, this collection doesn't have it yet.