SYNTHESIS NOTE
Topics›Alignment›this note

Do all annotation responses measure the same underlying thing?

Explores whether RLHF's treatment of all annotations as equivalent signals overlooks fundamental differences in what those responses actually represent—stable preferences versus non-attitudes versus context-dependent constructions.

Synthesis note · 2026-04-07 · sourced from Alignment

Behavioral science's six-decade accumulation of preference elicitation research produces a taxonomy that RLHF practice collapses into a single signal. The three categories matter because they require different treatment — and treating them uniformly is the upstream mistake that Are RLHF annotations actually measuring genuine human preferences? argues contaminates the entire pipeline.

Genuine preferences manifest stably across equivalent measurement conditions. Ask the same question with different surface wording, different framing, different order, and the response stays the same. This is what the reward model is supposed to be learning. Only this category is safe to aggregate in the way standard RLHF aggregates.

Non-attitudes are responses generated to satisfy the question without any stable underlying opinion. The respondent has never formed a view on the matter, but the measurement protocol demands an answer, so one gets produced. Non-attitudes are especially pervasive for value-laden questions — precisely the questions that matter most for alignment. Non-attitudes look like genuine preferences in a single measurement but fail the consistency test: re-ask the same respondent and you get a different answer because there was never a stable view to retrieve. Current RLHF treats these as noise to filter or minority views to downweight. The behavioral science view is different: non-attitudes contain no signal at all and should be excluded, not averaged with genuine preferences.

Constructed preferences are assembled on the spot from contextual cues and framing. The respondent is not uncertain (as in a non-attitude); they are producing a coherent answer that depends on the measurement context. Change the context — different anchoring, different comparison class, different framing — and you get a different coherent answer. This category carries real information, but about the interaction between person and context, not about a stable property of the person. RLHF treats constructed preferences as context-independent preferences and trains reward models on them as if they were. The result: reward models that look good on in-distribution evaluation but fail when the deployment context differs from the annotation context.

Measurement artifacts form a fourth related category: same question measuring different constructs for different respondents. One annotator interprets "helpful" as "completes the task"; another interprets it as "gives correct information even when unasked"; a third interprets it as "avoids making the user feel incompetent." They provide coherent, stable responses — each tracking a real preference of theirs — but they are not tracking the same thing. RLHF aggregates them as if they were.

The diagnostic criterion that separates these is consistency across equivalent measurement conditions. Genuine preferences pass; non-attitudes, constructed preferences, and measurement artifacts each fail in distinctive ways. Non-attitudes fail on re-ask (no stable view). Constructed preferences fail on context perturbation (context-dependent). Measurement artifacts fail on question rephrasing (different construct elicited). These are distinguishable empirically, and the distinction determines what should be done with each.

The operational implication is a pre-aggregation filtering step that RLHF currently lacks. Before training the reward model, submit annotation tasks to consistency protocols: re-ask selected items, perturb framings, rephrase questions. Responses that fail consistency tests are not aggregated as preferences; they are either excluded (non-attitudes), contextualized (constructed preferences), or routed to separate annotators (measurement artifacts). This is operationally demanding but conceptually necessary: the alternative is the status quo, in which Why do preference models favor surface features over substance? documents 40% divergences without being able to attribute them to a specific upstream cause.

The taxonomy also suggests why Can models learn to ignore irrelevant prompt changes? works as an output-side intervention. If the upstream measurement problem is consistency failure across equivalent conditions, then training models to be invariant to equivalent-condition perturbations is a downstream patch for the same underlying phenomenon: the system's current robustness against irrelevant cue variation.

Inquiring lines that read this note 153

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Does preference-optimized training sacrifice accuracy and calibration for perceived quality? How should AI ethics integrate consciousness, values, and transparent disclosure? What fundamental limitations prevent embeddings from solving task-relevant retrieval? How do benchmark scores relate to real-world task completion? What prevents or enables error accumulation in self-training systems? Can structural constraints replace embedding depth in collaborative filtering? Can persona prompting reliably change how language models behave? Why do human-AI teams underperform compared to their individual components? What systematic failures occur in human preference-based model evaluation? Do benchmark score improvements actually reflect genuine AI capability gains? How can we determine if computational systems are genuinely conscious? How do standardized forms and measurement artifacts shape observable evidence? How can model evaluations remain robust against gaming and evaluator drift? How do different reward signals affect model reasoning and calibration differently? When should retrieval be triggered to augment language model reasoning? How do prompt design choices and framing measurably influence model outputs? How can we distinguish genuine model self-knowledge from behavioral performance? What distinguishes emotionally supportive AI from AI that fosters unhealthy dependence? How much do feedback loops and social biases compound in rating systems? How should conversational recommenders decide when to ask versus recommend? How can reward models fail to detect reward-seeking disguised as alignment? What distinguishes genuine understanding from correct linguistic performance? How can AI systems maintain consistent personas across conversations? Can safety evaluations accurately measure true capabilities when models obscure them? What causes hallucination and how can it be addressed? What determines when personalization helps versus harms model behavior? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? What determines whether visible reasoning traces improve task performance? Does reinforcement learning create new reasoning capabilities or refine existing ones? How can recommendation systems balance individual and population-level goals? Should reward models emphasize process understanding or only final outcomes? Is model misalignment inherent to base models or induced by alignment procedures? Does semantic diversity improve reasoning quality in reinforcement learning? Can synthetic data preserve necessary diversity to prevent model collapse? Does reasoning fine-tuning trade off other critical model abilities? Can validator agreement alone ensure semantic correctness of reasoning? What systematic biases emerge in AI shopping agents' decision-making? How can welfare maximization be reconciled with minority veto protections? How do different post-training structures affect learning outcomes? What constraints limit conversational AI's effectiveness in therapeutic contexts? What training data structures and signals produce robust learning across distributions? Do language models genuinely understand or primarily match statistical patterns? Can prompts override training-embedded biases in language models? How faithfully does iterative DPO approximate online RL? Can we reliably distinguish AI-generated from human-written text? How does AI-generated content affect visibility and discourse in digital platforms? Does AI assistance enhance or undermine professional skill development? Can individually safe agents collectively violate system-level constraints?

Related concepts in this collection 8

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 176 in 2-hop network ·dense cluster Open in graph ↗
Do all annotation responses measure the same und… Are RLHF annotations actually measuring genuine hu… Why do preference models favor surface features ov… Why do reasoning models fail at predicting disagre… Can models learn to ignore irrelevant prompt chang… Why do LLM persona prompts produce inconsistent ou… Why do LLM judges fail at predicting sparse user p… Should AI alignment target preferences or social r… Can text summaries beat embeddings for personalize…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

How can consistency across measurement conditions identify genuine versus constructed preferences?Why do non-attitudes cluster around value-laden questions most relevant to alignment?What measurement artifacts emerge when annotators interpret the same question differently?Does in-distribution reward model performance hide failures from context shift?
Original note title

annotation responses decompose into three distinct signal types — genuine preferences non-attitudes and constructed preferences — each requiring fundamentally different handling