How does spending offline compute affect wake-time prediction latency?

This explores how compute spent offline — during training or precomputation — changes what you pay at the moment of prediction, when the system 'wakes up' to answer. The corpus frames this as a budget you can shift between two pools that turn out not to be independent. Can inference compute replace scaling up model size? is the cleanest statement of the tension: on hard prompts, a smaller model given more inference compute can match a larger one, which means pretraining and inference compute trade against each other rather than each doing their own job. So in principle you can move work earlier (bigger/better-trained model) to spend less at wake time, or move it later (more thinking tokens) to keep the model small.

But the corpus pushes back on the idea that the trade is symmetric. Can non-reasoning models catch up with more compute? finds that what you bake in offline can be qualitatively decisive — a model trained with a reasoning protocol stays ahead no matter how much inference budget a non-reasoning model is handed, because training is what makes those extra wake-time tokens productive in the first place. Can training data augmentation match test-time compute scaling benefits? shows the same move from the other direction: by spending offline compute to generate reasoning traces and fold them into pretraining, you get 3x data efficiency and harder tokens automatically absorb longer traces — essentially relocating test-time scaling into the training phase so the model arrives already 'pre-thought.'

The sharpest lens on the prediction-latency side is the internal-versus-external split in How do internal and external test-time scaling compare?: internal scaling (training models to reason autonomously) builds capability offline, while external scaling (search, verification, sampling at inference) extracts performance at wake time but adds latency. They complement rather than compete. And Does the choice of reasoning framework actually matter for test-time performance? adds a humbling caveat — once you're spending at inference, the specific algorithm (best-of-N vs MCTS) barely matters; total compute and the quality of your value function dominate. So the offline question becomes: did you train a good enough verifier/reward signal to make wake-time search worth its latency?

There's also a hardware-level version of this trade that's easy to miss. Does recomputing weights cost less than moving them on mobile? shows that on memory-bound mobile devices, recomputing a transformer block twice is actually *faster* than fetching separate weights — the bottleneck at wake time is moving data, not doing math. That inverts the usual intuition: spending more compute at prediction time can lower latency when memory movement is the real cost. The precomputation limit shows up vividly in How can real-time recommendations stay responsive and reproducible?, where Netflix can't precompute anything that depends on signals arriving mid-session, forcing runtime recomputation that raises call volume and timeout risk — a concrete case where you simply *can't* move the work offline.

The thing you didn't know you wanted to know: offline compute doesn't just 'speed up' wake-time prediction — it changes whether wake-time spending pays off at all. Training installs the protocol that makes inference tokens productive (Can non-reasoning models catch up with more compute?), and the deepest answers happen when both pools are tuned together rather than traded one-for-one.