Instead of just grading what an AI says, can you train it using the hidden numbers it computes along the way?
What does leveraging internal representations during training actually mean operationally?
This explores what it concretely means to use a model's hidden activations (the numbers flowing through its layers) as something you train with or against, rather than judging only its final text output.
This explores what it concretely means to use a model's hidden activations, the numbers flowing through its layers, as something you train with or against, rather than judging only its final text. The corpus suggests the phrase covers three different mechanics, and one of them is still mostly a proposal.
The most literal version is editing activations directly. Can editing hidden representations beat weight updates for finetuning? freezes all of the model's weights and learns a small intervention applied to the hidden states. Its low-rank variant (LoReFT) outperformed LoRA on reasoning, instruction-following and language-understanding benchmarks while using 10-50x fewer parameters. What gets trained is a small edit to the model's work-in-progress, not the model itself.
The second version is reading activations and using what you read. Can we decode what LLM activations really represent in language? trains a decoder to answer plain-language questions about what an activation encodes. That same decoder can then be used to steer the activations via gradient descent, so reading becomes control. The safety-flavored recipe is described in Can reward hacking vectors survive training-time use as detectors?. You compute a vector that separates reward-hacking behavior from honest behavior, then use it as a detector while the policy trains. This is the piece that is still unproven. The note says no one has shown whether a policy trained against that signal stops hacking or just learns to hide it from the detector.
The reason to look inside at all is that outputs can mislead. Can identical outputs hide broken internal representations? shows that networks with identical outputs can have very different internal structure. The fractured ones fail to transfer to new contexts or recombine ideas. What really happens inside a language model? pulls the same thread: matching benchmark scores can hide real trade-offs, where improving helpfulness or accuracy tends to erode faithfulness or calibration. Asking the model to describe itself doesn't fix this. Can language models actually introspect about their own states? finds that self-reports mostly echo training data, with genuine introspection only where a causal chain connects an internal state to the report. Reading activations directly bypasses the model's own narration.
The third meaning is looser. Here 'internal' means moved inside the model, not read from its activations. Can models learn to evaluate their own work during training? uses the unused space after the end-of-sequence token to train the model to score its own answer. The evaluator ends up in the weights and costs nothing at inference. Can models learn to internalize search algorithms through training? trains on linearized search traces so the model runs the search itself. Neither touches activations, so they change where a process lives, not what is inspected. Keep them apart when reading papers, because the same word is doing different work. Does instruction tuning teach task understanding or output format? is a caution for all three: what a model absorbs in training can be surface format rather than the understanding you hoped for, so any claim of 'internalized' needs a check on what was actually learned.
Sources 9 notes
ReFT learns task-specific interventions on frozen model representations rather than updating weights, with LoReFT (low-rank linear subspace variant) dramatically outperforming LoRA across reasoning, instruction-following, and NLU benchmarks while using far fewer parameters.
LatentQA trains a decoder to answer natural language questions about LLM activations, enabling both interpretability (understanding what activations encode) and controllability (steering them via gradient descent). Critical design choices—activation masking, diverse training data, and faithful completions—proved essential for generalization.
The paper identifies internal representations as a promising direction for training-time mitigation of reward hacking but provides no experimental evidence of whether this approach works. The key limitation is that no test shows whether a policy trained against the vector's signal still exhibits detectable hacking.
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
Research into mechanistic interpretability, cognitive models, and training dynamics shows that identical benchmark performance conceals radically different internal structures. Improving one capability (helpfulness, accuracy) reliably degrades others (faithfulness, calibration, diversity).
Show all 9 sources
LLM self-reports usually reflect human training distributions rather than actual internal processes. However, when a causal chain connects an internal state to accurate reporting—like inferring low temperature from output consistency—genuine lightweight introspection occurs without requiring consciousness.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
Meta-CoT demonstrates that instruction-tuning on linearized MCTS and A* traces teaches models to implement search strategies internally. This enables optimization over algorithms themselves rather than specific outputs, potentially unlocking novel reasoning strategies.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Semantic Structure in Large Language Model Embeddings
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning
- Post-Completion Learning for Language Models