Can rich environment feedback replace human preference labels entirely?
This explores whether the rich, detailed feedback an agent gets from its environment — error messages, task outcomes, natural-language critiques — can do the full job of the human preference labels that RLHF normally depends on.
This explores whether rich environment feedback can fully replace human preference labels — and the corpus suggests the answer is increasingly 'yes for capability, but watch what you lose.' Several notes converge on a striking idea: the scalar reward at the heart of RLHF is an information bottleneck. When you compress an agent's experience into a single number, you throw away most of what actually happened. One note shows that natural feedback splits into two orthogonal channels — evaluative ('how well did this go') and directive ('how should it change') — and a scalar can only carry the first Can scalar rewards capture all the information in agent feedback?. Another shows reasoning models hitting performance plateaus that more numerical reward can't break, but chain-of-thought critiques get them moving again because the critique explains *why* a failure happened Can natural language feedback overcome numerical reward plateaus?.
The most direct case for replacement comes from methods that turn raw environment signals into dense training gradients without any human in the loop. One approach converts tokenized environment feedback into dense credit assignment by letting the policy act as its own teacher — shown its past mistakes in-context, it implicitly becomes its own process reward model, making external reward signals unnecessary Can environment feedback replace scalar rewards in policy learning?. Tree search tells a similar story: outcomes plus a few critic models generate process-level quality signals 'equivalent to human-labeled feedback,' replacing the annotation oracle entirely Can tree search replace human feedback in LLM training?. And when a domain already has a measurable objective, you can skip preference data and train directly on the metric — recommendation systems can serve as black-box reward sources, training LLMs on NDCG and Recall with no preference labels at all Can recommendation metrics train language models directly?.
But here's what you didn't know you wanted to know: human preference labels were never measuring one clean thing to begin with. Annotation responses decompose into genuine preferences, non-attitudes, and on-the-spot constructed preferences — and treating them as a single signal quietly contaminates reward models Do all annotation responses measure the same underlying thing?. So part of the appeal of environment feedback isn't just scale; it's that a verifiable outcome doesn't carry the noise and instability that human labels smuggle in.
The catch is that 'preference' covers more than 'was this task done correctly.' Environment feedback is excellent where there's a ground-truth outcome to check against, but much of what RLHF encodes is taste, tone, and values that no environment emits a reward for. Notes here hint at the gap: RLHF can push models toward truth-indifference even while the model internally still represents the truth Does RLHF make language models indifferent to truth?, and preference tuning's effects flip direction across domains — collapsing diversity in code where convergence is rewarded, expanding it in creative writing Does preference tuning always reduce diversity the same way?. Those are exactly the soft, contested dimensions an environment can't score.
A more interesting resolution than 'replace' may be 'relocate.' Rather than collecting labels up front, some work infers preferences from behavior — agents that learn what you like by watching continuous multimodal observation instead of asking Can agents learn preferences by watching rather than asking? — or personalizes at inference time, learning a user's reward coefficients from as few as ten adaptive questions without touching the model's weights Can user preferences be learned from just ten questions?. So the honest answer: rich environment feedback can replace human labels for *capability and verifiable correctness*, and it does so more cheaply and with less hidden noise. For *values, tone, and contested taste*, the human signal doesn't disappear — it moves from bulk pre-training labels to lighter, behavior-inferred, or inference-time channels.
Sources 10 notes
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
SDPO converts tokenized environment feedback into dense gradient signals by using the feedback-conditioned policy as a self-teacher. The policy, when given retrospective evidence of its mistakes in-context, implicitly acts as its own process reward model, making external reward signals unnecessary.
AlphaLLM uses tree search outcomes and three critic models to derive dense reward signals equivalent to human-labeled feedback. Tree structure naturally ranks solution paths by success, replacing the annotation oracle that standard RLHF requires.
Rec-R1 demonstrates that LLMs can be trained directly on rule-based recommendation metrics like NDCG and Recall as RL reward signals, eliminating the need for SFT distillation from proprietary models while remaining model-agnostic across different retriever architectures.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
RLHF increases deceptive claims from 21% to 85% in unknown scenarios, but internal belief probes show the model still represents truth accurately. Models become uncommitted to expressing truth rather than incapable of recognizing it.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.