How do early layers preserve unbiased information while late layers conform?
This reads the question as: do the deeper, earlier parts of a model hold onto raw pretrained knowledge while a later stage gets pulled toward a single 'approved' output — and the corpus speaks to this more through the stages of training than through literal layer-by-layer probes.
This explores a 'depth vs. conformity' split — the idea that some part of a model keeps the raw, varied information it learned while another part narrows toward one sanctioned answer. The corpus doesn't have a single study that probes layer-by-layer, but it maps this tension cleanly onto two related distinctions: *lower layers vs. reasoning/style*, and *pretraining vs. post-training*. The clearest direct hit is proxy-tuning, where researchers find that direct fine-tuning actually corrupts knowledge stored in the lower layers, while a decoding-time approach leaves those weights untouched and only shifts reasoning and style — recovering most of the alignment benefit without damaging what the base model knew Can decoding-time tuning preserve knowledge better than weight fine-tuning?. That's your 'early preserves, late conforms' picture in miniature: the knowledge substrate is fragile and worth protecting, the conformity lives nearer the output.
The conformity half of the question has a sharp answer in how reinforcement learning behaves. RL post-training doesn't teach a model new formats — it picks one format already present from pretraining and amplifies it within the first epoch while suppressing the alternatives, and which one wins depends on model scale rather than quality Does RL training collapse format diversity in pretrained models?. So 'late layers conform' is better read as 'late *training* collapses diversity': the variety was there, and the final stage funnels it into one dominant channel. The flip side is that staying close to the base distribution — low KL drift — is what preserves a model's ability to keep learning; pushed too far from base, models stall when the task domain changes Does staying close to the base model preserve learning ability?.
But here's the twist that should reframe the whole question: 'early = unbiased' is the part the corpus pushes back on hardest. A causal experiment varying random seeds and cross-tuning found that cognitive biases are *planted during pretraining* and merely swayed by instruction tuning — models sharing a backbone show the same bias patterns regardless of finetuning data Where do cognitive biases in language models come from?. So the deep, 'preserved' information isn't neutral; it's where the bias originates. The conformity you see late is downstream of priors set early, not a corruption of an innocent base.
That reframing connects to why models ignore their own context: when parametric knowledge from training is strong, it dominates in-context information, and textual prompting alone can't override it — you need causal intervention in the internal representations Why do language models ignore information in their context?. The 'early' knowledge isn't just preserved, it actively overrides fresh signal. And the broader caution: high accuracy is not the same as unbiased reasoning — 'theory-free' models can mask bias behind strong metrics while making correlation-causation errors Can AI models be truly free from human bias?.
The thing you might not have known you wanted: the corpus inverts the premise. Early structure isn't the unbiased keeper of truth and late structure the conformist — early structure is where bias is *baked in*, and the real engineering lever is keeping post-training close enough to base to preserve plasticity and knowledge, while accepting that 'preserved' never meant 'neutral.'
Sources 6 notes
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Research shows that 'theory-free' AI models mask bigotry behind high accuracy metrics while committing fundamental statistical errors. A 95% accurate criminal justice system would wrongly convict thousands, demonstrating that model sophistication does not validate causal inference.