Should LLM teams train smaller models for longer?

The excerpt modifies the Chinchilla scaling laws — the empirical formulas that estimate how model quality changes with parameter count and training data — to account for the compute and dollar cost of inference, not just training. It argues that "LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal." The motivating observation: "the Llama 2 family of models were trained on 2 trillion tokens and the Llama 3 family of models were trained on 15 trillion tokens, which is far more data than the Chinchilla scaling laws would deem 'optimal.'" Since smaller models are cheaper to run, the extra training compute spent going past Chinchilla-optimal "pays off after enough inference requests."

The mechanism is a reformulation of the Chinchilla loss function L(N, Dtr) — parameterized by parameter count N and pre-training tokens Dtr — to solve jointly for the parameter and token counts that minimize combined training and inference compute (Sec. 2), and separately combined training and inference dollar cost (Sec. 6), given an assumed total inference demand Dinf (the sum of input and output tokens across all requests). The derivation assumes "that conditioned on pre-training loss, inference demand is independent of model size and token count" — i.e., models of equivalent quality but different sizes see the same request volume, so the only lever left is to shrink the served model and make up quality by training longer.

This modifies the same underlying scaling law as Can architecture choices improve inference efficiency without sacrificing accuracy?, but on a different axis. That note adds architectural variables (hidden size, MLP-to-attention ratio, GQA) as a lever that improves inference efficiency independently of parameter count or training compute; this excerpt instead treats the tokens-per-parameter ratio itself as the lever, holding architecture fixed, and ties the optimal ratio directly to projected inference volume. Read together, both reject the idea that a single training-compute-optimal model exists — the "optimal" model depends on how it will be architected and how much it will be asked to serve.

The excerpt's own inference-demand independence assumption is a simplification it flags, not a finding: Sec. 6's dollar-cost model "assume[s] MFU and cost per FLOP do not depend on model size, configuration, or sequence length," and leaves aside latency requirements. It also treats inference demand as a known, fixed input rather than something that itself depends on model price or latency in deployment — a feedback loop (cheaper, faster models may draw more usage) the excerpt does not model. The implication it does support, carefully stated: for any team that can forecast inference volume in the ~1B-request range, Chinchilla-optimal sizing is the wrong target, and the error compounds as projected demand grows.