When you ask people what they prefer, are you hearing a real opinion or one invented on the spot?
Can elicited user responses measure true preferences or just elicitation artifacts?
This explores whether what people say when asked (ratings, votes, critiques, answers to questions) reflects something stable about them, or is partly produced by the act of asking.
This explores whether what people say when asked reflects something stable about them, or is partly produced by the act of asking. The corpus's sharpest answer is that it's both, and the two can be separated. Do all annotation responses measure the same underlying thing? borrows from behavioral science to split responses into three kinds: genuine preferences, non-attitudes (the person had no real view but answered anyway), and constructed preferences (formed on the spot, shaped by how the question was framed). The tell is consistency across measurement conditions. Ask the same thing in a different way, and a genuine preference stays put while the other two move. Treating all three as one signal contaminates reward model training.
There's downstream evidence that models pick up artifacts. Do reward models actually consider what the prompt asks? finds that standard reward models learn response-level biases rather than whether a response answers the prompt, so they reward text that is well-written but irrelevant. The notes don't link this to the three-signal split directly, but the two fit together. When a rater has no strong view, surface polish is an easy thing to vote on. Framing sensitivity also isn't only human. Does emotional tone in prompts change what information LLMs provide? shows GPT-4 giving the same question different answers depending on emotional tone. So if simulated or LLM-mediated respondents are part of the pipeline, the artifacts can stack.
Elicitation isn't hopeless, though. Can crowdsourced votes reliably rank language models? shows 240K+ Chatbot Arena votes agreeing with expert raters, which suggests the signal is far from noise when the questions are diverse and discriminating. Agreement with experts shows the votes are consistent, not that they capture any one person's true preference. And Can aggregate reward models satisfy genuinely disagreeing users? points to a different artifact that appears even when responses are genuine. Averaging a 51-49 split means either 49% are always unhappy or everyone is unhappy half the time. Here the distortion comes from aggregation, not from asking. One way around it is to ask in a smarter, person-specific way. Can user preferences be learned from just ten questions? shows that about ten adaptively chosen questions can pin down an individual's reward coefficients, though this assumes preferences are stable enough to be a combination of shared base rewards.
The other route is to stop treating the raw answer as the truth. Can language models bridge the gap between critique and preference? shows that a remark like "doesn't look good for a date" is an indirect window onto a preference, and rewriting it as "prefer more romantic" retrieves better matches. Does abstract preference knowledge outperform specific interaction recall? finds that abstracted preference summaries beat recalling specific past interactions, which suggests the durable signal lives in the pattern across responses and not in any single one. Can agents learn preferences by watching rather than asking? goes furthest, inferring preferences from continuous observation without asking at all.
So elicited responses can measure true preferences, but only some of them do, and a single response can't tell you which. The practical lesson from these notes is to check consistency across framings before trusting an answer, keep individuals distinguishable, and treat what people say as evidence about a preference. The corpus has no head-to-head study of asked versus observed preferences for the same people, so how far observation actually escapes these artifacts is still open here.
Sources 9 notes
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Standard reward models learn response-level biases instead of prompt-response alignment, causing them to reward responses that are well-written but irrelevant. Decomposing reward into prompt-free and prompt-related components reveals this failure and enables targeted fixes.
GPT-4 exhibits emotional rebound (negative prompts yield ~86% neutral-positive responses) and a tone floor (positive prompts rarely go negative), causing identical questions to receive different answers depending on emotional framing. This bias is suppressed only on sensitive topics where alignment constraints override tone effects.
Chatbot Arena's 240K+ crowdsourced preference votes produce credible model rankings because the underlying questions are diverse and discriminating, and crowd judgments correlate with expert raters—validating human preference as a scalable evaluation signal.
Single reward models trained on aggregated preferences cannot represent disagreement. A 51-49 preference split forces a choice between leaving 49% unhappy always or leaving everyone unhappy half the time. This is a representational failure, not a quality problem.
Show all 9 sources
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
Few-shot LLM prompting can convert natural negative feedback like "doesn't look good for a date" into positive preferences like "prefer more romantic," enabling retrieval systems to find better-matching recommendations without fine-tuning.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Measuring Human Preferences in RLHF is a Social Science Problem
- Preference Discerning with LLM-Enhanced Generative Retrieval
- Capturing Individual Human Preferences with Reward Features
- Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
- PRIME: Large Language Model Personalization with Cognitive Memory and Thought Processes
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Information-Theoretic Reward Decomposition for Generalizable RLHF
- Evaluating the Hidden Costs of Personalization in Large Language Models