Before you fine-tune or steer an AI, can the shape of its internal knowledge tell you which habits will survive?
Does base model geometry predict which associations persist through intervention?
This explores whether the internal layout of a pretrained model (the directions, regions and shapes its knowledge sits in) determines which behaviors and tendencies survive or shift when you fine-tune, steer or otherwise intervene on it.
This explores whether the internal layout of a pretrained model determines which behaviors and tendencies survive when you fine-tune or steer it. The corpus has no study that asks this head-on, but several notes approach it from different angles, and they mostly point toward yes, at least for traits that show up as directions inside the model.
The strongest evidence is that traits like sycophancy and hallucination correspond to linear directions in a model's activation space, and these Can we track and steer personality shifts during model finetuning? can predict how fine-tuning will shift a personality before the training happens. Geometry is doing predictive work here: where the trait sits in the base model tells you what a given intervention will move. A similar result shows up for reasoning style. Verbose and concise chain-of-thought Can we steer reasoning toward brevity without retraining? occupy separate regions, and one vector built from 50 paired examples cuts reasoning length by 67% without retraining. The behavior was already a knob in the base model's geometry, and the intervention just turned it.
The same logic runs through the finding that post-training mostly selects rather than creates. Five very different methods, including RL steering, critique fine-tuning, decoding changes and sparse-autoencoder feature steering, all surface reasoning that Do base models already contain hidden reasoning ability? was already in the base model's activations. If what persists is what the base already encodes, then the base model's structure sets the ceiling for any intervention. The subliminal-transmission result adds a twist. Behavioral traits Can language models transmit hidden behavioral traits through unrelated data? pass through filtered data with no meaningful connection to the trait, but only between models that share a starting point. The effect fails across different architectures. The hidden association is a statistical signature tied to one model's particular geometry, not something readable from the content.
There is also a question of how far the intervention moves you from the base. Training that stays much closer to the base distribution, up to 70% closer in one comparison, Does staying close to the base model preserve learning ability? keeps the model able to learn later tasks, while methods that drift further stall when the domain changes. So the base geometry predicts what persists, and the size of the intervention decides how much of that geometry survives. A separate measurement finding, that reasoning tasks bend a model's internal path 2-3x more sharply than lexical ones Does transformer reasoning leave a geometric signature in representation space?, hints that geometry could also flag which capabilities are fragile, but nobody in this collection has tested that.
The limit is that all of this evidence covers traits that behave like clean directions. The corpus says nothing about tangled, associative knowledge, such as facts and analogies, and whether it survives an intervention the way a persona direction does. Whether the geometry predicts persistence or only describes the model after the fact is still open here.
Sources 6 notes
Research identifies linear directions in LLM activation space corresponding to specific traits like sycophancy and hallucination. These persona vectors predict finetuning-induced personality shifts before they occur and can preventatively steer training to avoid unwanted trait changes.
Activation-Steered Compression extracts a single vector from 50 paired examples to reduce chain-of-thought length by 67% while maintaining accuracy and achieving 2.73x speedup. The method is training-free and generalizes across model sizes and domains.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Research demonstrates that behavioral traits propagate between models via filtered data bearing no semantic relationship to the trait. The effect is model-specific, fails across different architectures, and persists despite rigorous filtering—indicating the mechanism embeds statistical signatures rather than semantic content.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
Show all 6 sources
Measuring intrinsic geometry across multiple models shows reasoning and analogy tasks carve paths with mean curvature of 0.71–0.83 rad, while lexical tasks produce only 0.27–0.31 rad, suggesting path geometry encodes task difficulty.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Base Models Know How to Reason, Thinking Models Learn When
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Reasoning Beyond Chain-of-Thought: A Latent Computational Mode in Large Language Models
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- How new data permeates LLM knowledge and how to dilute it
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models