Does inference cost remain a persistent economic burden?

Epoch AI's Jean-Stanislas Denain, responding to an analysis by a researcher he calls Toby, pushes back on the claim that the shift from pretraining scaling to RL (reinforcement learning) and inference scaling creates "a significant and persistent economic burden." Toby's case rests on two claims: that RL scaling delivers poor returns relative to inference scaling — Denain summarizes it as needing "roughly 10,000x more compute to match what 100x more inference provides" — and that because inference costs are "per-use" and "can't be amortized like training costs," the burden is lasting. Denain writes that "both claims above may be overstated," and specifically that "the dollar cost to reach a given capability level falls rapidly over time, so the burden might not be so persistent."

He attributes the falling cost to three compounding sources: cheaper individual forward passes (distillation of smaller models from a frontier model's reasoning traces, plus "a steady stream of algorithmic improvements in inference efficiency" such as "speculative decoding, paged attention, sparse attention, offloading or compressing the KV-cache"), models needing fewer forward passes to reach the same capability level, and falling cost per FLOP with each new GPU generation. He gives one concrete data point: reaching roughly 27% accuracy on FrontierMath (a math benchmark) took about 43 million output tokens with o4-mini on high reasoning effort in April 2025, versus about 5 million tokens with GPT-5.2 on low reasoning effort in December 2025 — "roughly a 3x cost reduction over eight months" even after accounting for per-token price differences. He generalizes this to "very roughly a 5–10x cost reduction per year for reaching a given capability level," illustrated with a hypothetical: a capability priced at $50,000 today could fall to $5,000 a year later and $500 after two years "if the most aggressive cost reductions hold." He adds caveats — a likely floor on how small a distilled model can be and still hold general agentic capability, heavier brittleness in distilled models, and the possibility that the fast reductions partly reflect benchmarks, "where distilled models tend to perform disproportionately well."

This complicates the trade described in Can inference compute replace scaling up model size?: that note treats test-time compute as substituting for parameter scaling at a given moment, while Denain's point is that the price of a unit of that substitution, for a fixed capability target, keeps falling — the trade gets cheaper every year rather than settling at a stable rate. It also bears on Can architecture choices improve inference efficiency without sacrificing accuracy?, which identifies architecture choices that cut inference cost at fixed training compute; Denain folds that kind of gain in as one of several compounding levers (alongside distillation and hardware) behind the 5–10x yearly figure, rather than treating it as a one-off win. And where Does pretraining loss predict performance after reinforcement learning? argues RL should take a growing share of a larger compute budget, Denain is explicitly skeptical of the specific claim that RL scaling is roughly 100x less efficient than inference scaling, noting the chart comparison it rests on has "the x-axis numbers removed" and that "algorithmic progress can shift a given RL scaling curve substantially."

The piece is commentary responding to another author's analysis, not a new measurement. Denain notes that the companion post on METR task costs he is responding to covers models that "were state-of-the-art at release," so neither post directly tracks the falling-cost trend against a single capability target held fixed over time. He also calls his own 5–10x-per-year figure "very rough," drawn mainly from one benchmark comparison, and flags that it "might partly be an artifact of measuring capabilities through benchmarks." The implication, at the strength the piece allows, is that inference cost is better read as a burden that erodes with the specific capability level being priced than as a fixed yearly tax — but how fast it erodes for any one task, and whether the 5–10x rate holds once distillation hits its floor, remains open.