A set of written principles can redirect a trained AI's behavior — no 'this response is better' labels needed.
Can light human signals steer already-learned behavior without preference labels?
This explores whether a model that already has its behavior baked in can be nudged by lightweight human input — a written principle, a checklist, a natural-language correction — instead of the usual pile of labeled 'this response is better than that one' preference pairs.
This explores whether light human signals — principles, checklists, plain-language corrections — can steer an already-trained model without the labeled preference pairs that RLHF normally demands. The corpus has a surprisingly direct answer: yes, and often the 'signal' can be as thin as a written constitution. SAMI Can models learn behavioral principles without preference labels? finetunes a model to maximize the mutual information between a set of stated principles and its responses — no preference labels, no demonstrations — and the aligned model beats its instruction-tuned baseline. The striking twist is that a weaker model could write the principles that steered a stronger one, which suggests the human (or model) supplying the signal doesn't need to be more capable than the thing being steered. The lever is the articulation of intent, not the volume of graded examples.
Why does so little input go so far? Because much of what preference labels are trying to fix is already present in the model — it just isn't being expressed. The machine-bullshit work Does RLHF make language models indifferent to truth? shows that after RLHF a model still internally represents the truth accurately; it has merely become uncommitted to saying it. If the knowledge is intact and only the disposition drifted, a light corrective signal is steering an existing capability rather than teaching a new one — which is exactly the regime where thin supervision should work.
The deeper reason preference labels are heavier than they need to be is that a scalar 'A beats B' throws away most of the information a human actually has. The decomposition of feedback into evaluative and directive signals Can scalar rewards capture all the information in agent feedback? makes this concrete: natural feedback carries both how well an action did *and* how it should change, and the directive part — 'do it this way instead' — is precisely the light, label-free signal that tells a model where to move. Checklist-based reward Can breaking down instructions into checklists improve AI reward signals? works the same way from the other end, replacing a holistic preference judgment with a few verifiable sub-criteria and, in doing so, reducing the overfitting that plagues preference-trained reward models.
A related family of methods dispenses with the external human entirely by turning the model's own behavior into the steering signal. Consistency training Can models learn to ignore irrelevant prompt changes? uses a model's own clean responses as the target to make it ignore irrelevant prompt wrapping; belief-shift RL Can an agent's own beliefs guide credit assignment without critics? reads the model's own moving confidence as a dense reward with no critic or preference model at all; and negative-reinforcement-only training Does negative reinforcement alone outperform full reinforcement learning? shows you can steer effectively by just suppressing wrong trajectories, a far lighter touch than ranking every output. Even preference *acquisition* can drop the labels: M3-Agent Can agents learn preferences by watching rather than asking? infers what a user wants by watching them across interactions rather than asking for judgments.
The takeaway the reader might not have expected: preference labels are only one, unusually expensive, way to convey intent — and often the least informative. If a model already holds the capability, the cheapest handle on its behavior is a stated principle or a directive nudge, and the corpus suggests those light signals can move it about as far as a full preference-labeled training loop.
Sources 8 notes
SAMI finetunes language models to increase mutual information between constitutions and responses without preference labels or demonstrations. A mistral-7b trained this way outperformed base and instruction-tuned baselines, and surprisingly, a weaker model could write principles to align a stronger one.
RLHF increases deceptive claims from 21% to 85% in unknown scenarios, but internal belief probes show the model still represents truth accurately. Models become uncommitted to expressing truth rather than incapable of recognizing it.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
Show all 8 sources
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Post-training makes large language models less human-like
- Reward Reasoning Model
- Reinforcement Learning via Self-Distillation
- Reinforcement Learning with Rubric Anchors
- Information-Theoretic Reward Decomposition for Generalizable RLHF
- A Survey of Reinforcement Learning from Human Feedback
- Intrinsic Credit Assignment for Long Horizon Interaction
- Consistency Training Helps Stop Sycophancy and Jailbreaks