A model's weights get crushed to 4-bit — so how does it still learn anything when you fine-tune it?

This explores how a model stored in 4-bit precision can be fine-tuned without losing accuracy, even though training usually needs high-precision numbers. The short answer reframes the question: in the main approach the corpus covers, the 4-bit weights never actually learn anything. They stay frozen. Gradients pass *through* them during backpropagation, but the only parameters that get updated are small, full-precision Low-Rank Adapters (LoRA) attached alongside. That is how Can we finetune huge language models on a single GPU? cuts the memory for fine-tuning a 65B model from over 780GB to under 48GB and still matches full-precision fine-tuning results. The compressed weights only need to be accurate enough to carry the signal forward and backward. The precise, learnable part of the system is kept small and separate.

That split between a frozen, compressed base and a small trainable add-on turns out to be a recurring pattern, not just a memory trick. Can editing hidden representations beat weight updates for finetuning? goes further. It leaves the weights alone entirely and learns small edits to the model's internal activations, which reportedly uses 10–50x fewer parameters than LoRA. Can decoding-time tuning preserve knowledge better than weight fine-tuning? doesn't touch the model's internals at all. It steers only the output probabilities, and it finds that this *preserves* factual knowledge better than direct fine-tuning, which tends to corrupt knowledge stored in the lower layers. Read together, these suggest that freezing the base weights may be a feature as well as a workaround. Not touching them protects what the model already knows.

There's a second, quite different answer to how low-bit weights can stay accurate: train them that way from the start. Can ternary weights match full precision model performance? describes BitNet b1.58, whose weights take only three values (−1, 0, +1). Because the model is trained in this form natively, rather than compressed after the fact, it matches standard 16-bit models of the same size on benchmarks while using less memory and energy. QLoRA keeps a high-precision model's knowledge by not disturbing it. BitNet shows that a model can learn to work well within very low precision if it is trained under that constraint. There's also a hardware angle: Does recomputing weights cost less than moving them on mobile? shows that on phones, moving weights in and out of memory costs more than computing with them. That's a big reason compressing weights pays off so well in practice.

Where the corpus runs thin: it doesn't cover the specific techniques that make QLoRA's 4-bit storage hold up numerically, such as the 4-bit data type tuned to how weights are typically distributed, quantizing the quantization constants themselves, or managing memory spikes during training. If you want those mechanics, the QLoRA note is the doorway, but you'd need the original paper for the details.