A model's training, its study material, and your prompt each open or close different paths — which behaviors can you actually reach?
How do weights, selection, and prompts create different geometric landscapes of accessible behaviors?
This explores how three levers (a model's trained weights, the selection of which prompts and examples get trained on, and the prompt written at inference time) each shape which behaviors a model can actually reach, treating behavior as a landscape with terrain, ridges, and paths.
This explores how three levers (trained weights, selection of training material, and the prompt you write) each shape which behaviors a model can actually reach, treating behavior as a landscape you move through. The corpus has no single note that puts all three on one map, so what follows is stitched from separate findings. They fit together better than you might expect.
Start with the prompt, because it looks the most powerful and is the most constrained. In theory a single fixed transformer can compute any computable function if given the right prompt Can a single transformer become universally programmable through prompts?. In practice standard training rarely produces a model that uses prompts that way. So the set of behaviors a prompt could reach is far larger than the set the weights make easy to reach. Within that limit, prompt structure still matters: a single model prompted to play several personas can reproduce what a multi-agent system does Can branching prompts replicate what multi-agent systems do?. The geometry is also literal. Reasoning tasks trace paths through representation space with two to three times the curvature of simple lexical tasks Does transformer reasoning leave a geometric signature in representation space?. Verbose and concise chains of thought sit in different regions, separated by a direction you can find from about 50 paired examples. Nudging along that direction cuts reasoning length by 67% with no retraining Can we steer reasoning toward brevity without retraining?. The prompt and its activations move you around within terrain the weights already carved.
Selection decides which parts of that terrain training carves deeper. When the rewards within a prompt barely differ, the learning signal weakens and the model slides into generic, input-ignoring templates. Filtering training to high-variance prompts recovers performance Why do language models collapse into generic templates?. Reinforcement learning in search agents narrows behavior toward a few reward-maximizing strategies, while supervised training on diverse demonstrations keeps the range wide Does reinforcement learning squeeze exploration diversity in search agents?. What you keep matters as much as how much: training on messy trajectories with failures and backtracking teaches sturdier reasoning than training on clean shortcuts Can models learn better by training on messy exploration paths?. Handling successes as concrete demonstrations and failures as abstracted lessons also beats treating them alike Should successful and failed episodes be processed differently?. Selection sets whether the landscape stays a broad plain or narrows into a single ridge.
The weights are also shaped by what kind of signal reaches them. A scalar reward is a single number that says how well the model did, and it says nothing about which way to move. Feedback carries a second, directional component that scalars discard Can scalar rewards capture all the information in agent feedback?. This may explain why models stuck on a plateau can produce correct solutions once given a written critique Can natural language feedback overcome numerical reward plateaus?. The right behavior was reachable all along. The numerical signal just gave the model no slope to climb toward it. Whether a behavior is accessible therefore depends on more than whether the weights can represent it. It also depends on whether the training signal points toward it.
The three levers work on different timescales. Weights are slow and permanent and set the terrain. Selection chooses what gets to reshape it, which can widen or flatten it. Prompts and activation steering are fast and reversible, and they move you around inside what the other two left. The outside world can be a fourth lever. Reorganizing a codebase around runtime behavior let weaker planners match stronger models at finding the right code Can explicit behavior maps help weaker planners compete with stronger models?. That reshaped the map the model reads rather than the model.
Sources 11 notes
Research proves a single finite-size transformer exists that can compute any computable function given the right prompt, achieving complexity bounds nearly matching unbounded models. However, standard training rarely produces models that learn to implement arbitrary programs this way.
Research shows single LLMs using dynamic persona simulation achieve multi-agent cognitive synergy without multiple model instances. Solo Performance Prompting validates that structured prompting techniques map directly to multi-agent debate architectures, enabling equivalent outcomes through structural equivalence.
Measuring intrinsic geometry across multiple models shows reasoning and analogy tasks carve paths with mean curvature of 0.71–0.83 rad, while lexical tasks produce only 0.27–0.31 rad, suggesting path geometry encodes task difficulty.
Activation-Steered Compression extracts a single vector from 50 paired examples to reduce chain-of-thought length by 67% while maintaining accuracy and achieving 2.73x speedup. The method is training-free and generalizes across model sizes and domains.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Show all 11 sources
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Research shows that training on messy trajectories—failed attempts, self-correction, and backtracking—teaches more robust reasoning than training only on shortcut solutions. This approach models o1-style deep reasoning as search internalization rather than solution memorization.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
- RAGEN-2: Reasoning Collapse in Agentic RL
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Efficient Reinforcement Learning via Large Language Model-based Search
- Teaching Large Language Models to Reason with Reinforcement Learning
- Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity