Why do optimal learning dynamics improve scaling law coefficients specifically?
This explores why the *way* a model is trained — its learning dynamics — shifts the constants inside a scaling law (moving the whole curve up), rather than simply requiring more data or parameters to climb the same curve.
This explores why optimizing how a model learns improves the *coefficients* of a scaling law — the multipliers and exponents that govern how loss falls as you add compute — instead of just buying more scale. The honest starting point: the corpus doesn't contain a single paper that proves this claim head-on, but several notes converge on the territory and explain the mechanism. The key reframe is that a scaling law isn't a law of nature. Its coefficients aren't fixed constants — they encode how efficiently a given training recipe converts compute into capability. Improve the dynamics and you change the constant, which looks like getting more out of the same budget.
The clearest lens comes from the idea that deep learning theory is consolidating around 'learning mechanics' — modeling training the way physics models gases, prioritizing average-case behavior, training dynamics, and aggregate statistics over worst-case bounds Can deep learning theory unify around training dynamics?. Under this frame, scaling coefficients stop being mysterious givens and become *outputs* of the trajectory a model takes through training. If the dynamics are the thing that produces the coefficient, then better dynamics — smoother optimization, less wasted movement, preserved capacity to keep learning — should produce a better coefficient.
You can see this concretely in work that folds architectural choices directly into the scaling law. By adding variables like hidden size, MLP-to-attention ratio, and attention grouping, models reorganize the *same* training budget into up to 2.1% higher accuracy and 42% more throughput than a strong baseline Can architecture choices improve inference efficiency without sacrificing accuracy?. Nothing was scaled up; the curve itself moved, because structural and training choices change the constant in front of it. That's the coefficient story in miniature.
The corpus also suggests *why* good dynamics matter so much: they protect the model's ability to keep improving. Staying close to the base distribution (low KL drift) preserves 'plasticity' — models trained that way keep adapting where parameter-only methods stall when the task shifts Does staying close to the base model preserve learning ability?. And reinforcement learning turns out to touch only 5–30% of parameters in structured, full-rank subnetworks rather than thrashing the whole network Does reinforcement learning update only a small fraction of parameters?. Both hint that 'optimal' dynamics are disciplined ones — they move the right things and leave capacity intact, which is exactly what keeps the scaling curve steep instead of flattening early.
The most surprising adjacent finding is that better learning schemes don't just shift one curve — they can open *new* scaling axes entirely. Latent-thought models couple fast local learning with slow global learning and get better sample and parameter efficiency, creating a scaling dimension independent of parameter count Can latent thought vectors scale language models beyond parameters?, and deep-research agents reveal a search-budget axis that scales like reasoning tokens do Do search steps follow the same scaling rules as reasoning tokens?. The takeaway a curious reader might not expect: 'improving the coefficients' and 'finding a new axis to scale on' are two faces of the same insight — that the dynamics of learning, not raw size, are what the scaling laws are quietly measuring.
Sources 6 notes
Research shows learning mechanics is consolidating as a unified frame for deep learning, modeled on classical and statistical mechanics. It prioritizes average-case predictions, training dynamics, and aggregate statistics over worst-case bounds, mirroring how physics addresses macroscopic systems.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Latent-Thought Language Models achieve superior sample and parameter efficiency by coupling fast local variational learning with slow global decoder learning. This dual-rate scheme scales few-shot reasoning across both model and latent size, creating independent scaling dimensions beyond traditional parameter scaling.
Deep research agents improve with more search steps in a pattern mirroring the reasoning-token relationship, with both exhibiting diminishing returns. This reveals a new inference-compute axis beyond model capability alone.