Why can an AI company's per-query profit look healthy while its overall books stay deep in the red?

This explores the gap between what an AI lab earns on each query it serves and what it keeps once it has paid for everything, including training the next model. One caveat first: no note in the collection reports gross and operating margin figures for model providers directly. What the corpus does have is the cost structure underneath those numbers, and that is enough to see why the two figures can be so far apart.

Start with the costs. Epoch AI's analysis of Anthropic, Minimax and Z.ai finds that compute makes up 54–62% of total operating expenses, while staff stay under 25% despite very high salaries Do compute costs really dominate AI company budgets?. The key detail is that this compute figure lumps together two kinds of spending that accountants put in different places. Inference compute, the cost of answering your prompts, is a cost of serving customers, so it counts against gross margin. Training and research compute counts as R&D, so it only shows up when you get to operating margin. A lab can make healthy money on each API call and still lose money overall, because the biggest single cost, building the next frontier model, sits below the gross-margin line.

Why is compute so expensive? Epoch's model of a one-gigawatt data center finds that servers account for about 60% of the roughly $8.5B annual cost, while electricity is only about $0.6B What actually drives AI data center costs?. So the main cost is paying off hardware, not running it. For a model provider, that means margins depend on keeping expensive chips busy. Idle capacity drags down margins whichever line it lands on.

The part you might not expect is that gross margins on any one capability don't stay put. Epoch AI also finds that the price of a given level of benchmark performance falls 66% per quarter right after it becomes state of the art. That slows to 32% per quarter within two years, as a short-lived monopoly gives way to competition Do AI performance prices fall faster when capabilities debut?. Labs earn their best per-query margins only during a brief window after launch. To keep those margins, they have to keep paying for new training runs, and that spending is exactly what pulls operating margin down. Gross margin depends on staying at the frontier, and staying at the frontier is what eats into operating margin. Work showing that compact, efficiently trained models can match much larger ones at a fraction of the cost Does model efficiency matter more than peak capability for real work? is one way out of that trap. It's also one more source of the competition that drives prices down.

For a similar gap one layer down the supply chain, look at AI training-data companies. Mercor's $2B in gross annualized revenue shrinks to roughly $600–800M once 60–70% goes to the experts it pays Can AI data companies sustain margins beyond labor payout ratios?. It's the same lesson: the headline number and the number a business actually keeps can be very far apart. Which costs land above or below the line shapes how profitable a company looks.