Counting AI's computing power looks like a checkable way to pace frontier AI, but does more compute still reliably mean more capability?

This explores why compute, meaning the chips and processing power that go into building and running AI, is often proposed as a measurable way to pace frontier AI. The basic idea is that compute is physical, countable and concentrated, so it seems easier to verify than something abstract like 'capability.' Be aware that this collection has no notes on compute governance itself: nothing on chip tracking, training-run thresholds or reporting regimes. What it does have is research that tests the assumption underneath the whole idea, which is that more compute reliably means more capability. That evidence is more complicated than the policy framing suggests.

The first problem is that capability is moving away from raw training scale. A 3B-parameter model reached frontier-level math and coding scores mainly through careful post-training design, not size Can small models match frontier reasoning without massive scale?. Related work finds that how a model is trained matters more than how much compute it uses at inference time Can non-reasoning models catch up with more compute?. If recipes matter as much as raw processing power, counting floating-point operations misses much of what drives progress.

The second problem is that compute is split between training and use, and the two can stand in for each other. Snell et al. showed that a smaller model given more 'thinking' compute at answer time can match a larger model on hard prompts Can inference compute replace scaling up model size?. Spreading that same compute adaptively, giving more to harder prompts, does better still Can we allocate inference compute based on prompt difficulty?. A rule that only counts training runs could miss capability gained after deployment. The same pattern now applies to reward models, which improve by reasoning before they score Can reward models benefit from reasoning before scoring?. Taken further, breaking a task into very small steps and voting on each one let small non-reasoning models complete million-step tasks without errors Can extreme task decomposition enable reliable execution at million-step scale?. How a system is organized can replace how big it is.

The third problem matters most for pacing: systems that improve themselves. The Darwin Gödel Machine improved its own coding-agent design by trial and error, more than doubling its SWE-bench score Can AI systems improve themselves through trial and error?. On long optimization tasks, success depends less on starting quality than on persistence, meaning how long an agent keeps running the benchmark-edit-retry loop What predicts success in ultra-long-horizon agent tasks?. In that setting, compute turns into wall-clock time spent iterating, which is a different thing to measure than a training run's size. The good news is that current research agents mostly recombine known techniques rather than invent new ones Do frontier AI agents actually conduct novel research or just optimize?.

What the reader might not expect is that verifying compute is the easy half. A compute limit only paces development if you can also check what capability a given amount of compute buys, and standard benchmarks both overstate and understate that. One proposed fix is open-world evaluations of messy, long tasks with costs reported explicitly Do automated benchmarks hide what frontier AI systems can really do?. That pairing is the real implication here: compute can be checked, but it only works as a pacing tool if capability evaluation keeps up. For the policy question itself, the collection would need sources on compute governance that it doesn't have yet.