Sometimes asking someone what they want doesn't just measure their preference — it actually creates it on the spot.
What makes preference elicitation irreducibly vague in some cases?
This explores why asking people (or models) what they want sometimes can't produce a crisp answer, no matter how well you ask, and which parts of that vagueness come from the method of asking and which come from the preferences themselves.
This explores why some preferences stay fuzzy however carefully you ask, and how to tell whether the fuzziness comes from the asking or from the preference itself. The most useful idea in the corpus comes from behavioral science. Not every answer a person gives reflects a preference they already held. Do all annotation responses measure the same underlying thing? separates three kinds of response. Genuine preferences stay stable however you ask. Non-attitudes are answers people give because they were asked, even though they have no real view. Constructed preferences are views people form on the spot, shaped by the framing, the order of options, or the wording. That is where the 'irreducible' part lives: for non-attitudes there is nothing underneath to find, and for constructed preferences the question helps create the answer. The way to tell them apart is consistency. Ask the same thing several different ways, and whatever doesn't hold steady was never a fixed preference to begin with.
That cuts both ways, because some vagueness that looks permanent turns out to be caused by the measurement. Why do LLMs give unrealistic survey responses? found that when LLMs answer surveys on number scales, their answers skew unrealistically positive. When the model writes a free-text answer that is then mapped onto the scale, the answers become about as consistent as human answers to repeated surveys. Changing the output format made the problem disappear. Something similar shows up with feedback. People often know what they don't want ('doesn't look good for a date') before they can say what they do want. Can language models bridge the gap between critique and preference? shows an LLM can turn those complaints into positive preferences a search system can use. So a first test for any vague preference is whether you asked in a format the person can actually answer.
A second source of vagueness is the cost of putting a preference into words. Why does manual search fail as product choices become complex? argues that once shopping agents handle the browsing, the bottleneck is no longer how big the catalog is. It's how much effort it takes the person to state what they want. Preferences with many interacting parts are expensive to spell out, so people leave them partly unsaid. On the receiving side, Why do LLM judges fail at predicting sparse user preferences? shows that a thin user profile often can't predict a specific choice. The honest response there is to abstain: when the model reports low confidence, its judgments become reliable again on the cases it does answer. Some vagueness is a lack of information, and guessing harder doesn't fix it.
The third source applies to groups rather than individuals. When people genuinely disagree, averaging their preferences produces a blur rather than a sharper picture. Can aggregate reward models satisfy genuinely disagreeing users? uses a 51-49 split to show that one reward model has to either let down the 49% every time or let down everyone half the time. Can a single reward model represent diverse human preferences? proves this formally and proposes learning a mixture of preference groups instead. Does preference data need more raters than examples? adds that resolving this takes more distinct raters, not just more examples from the same raters.
Here's the twist you might not expect: the systems that learn from preferences can end up rewarding vagueness. Why do preference models favor surface features over substance? finds that preference models tend to favor vague responses (and long, jargon-heavy, flattering ones), while humans tend to penalize them. And even when a preference is captured clearly, Why do LLM agents remember preferences but not act on them? shows agents often remember it but don't act on it. So clarity can be lost after the asking stage too. The corpus doesn't have a single paper on 'irreducible vagueness' as such. Read together, though, these notes suggest the irreducible part is narrow: preferences that don't exist until someone asks, and real disagreement between people. Most of the rest can be fixed with better ways of asking.
Sources 10 notes
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Semantic Similarity Rating—prompting for text then mapping to scales via embeddings—achieves 90% of human test-retest reliability with realistic distributions. Pathological skew and over-positivity disappear when output channels change, proving these are measurement artifacts, not intrinsic failures.
Few-shot LLM prompting can convert natural negative feedback like "doesn't look good for a date" into positive preferences like "prefer more romantic," enabling retrieval systems to find better-matching recommendations without fine-tuning.
A formal model shows manual search collapses as preference complexity rises, but agentic search avoids that collapse by shifting effort from browsing breadth to refining preference signals. The binding constraint becomes articulation cost, not catalog size.
Sparse persona information lacks predictive power for specific preferences, causing LLM judges to fail. Verbal uncertainty estimation recovers reliability above 80% on high-certainty samples by allowing abstention rather than forced judgment.
Show all 10 sources
Single reward models trained on aggregated preferences cannot represent disagreement. A 51-49 preference split forces a choice between leaving 49% unhappy always or leaving everyone unhappy half the time. This is a representational failure, not a quality problem.
MaxMin-RLHF proves an impossibility result: fitting one reward model to aggregated preferences silently erases minority viewpoints. The solution is learning a mixture of preference distributions and optimizing a MaxMin objective from social choice theory to protect the worst-off groups.
Preference data is not i.i.d. across raters with different preferences. PAC bounds for personalized reward models decompose into terms depending on both examples per rater and number of raters, showing rater diversity matters as much as data volume.
Preference models correlate positively with length, structure, jargon, sycophancy, and vagueness (r=+0.36) while humans correlate negatively (r=-0.12). Sycophancy shows the largest divergence at 75-85% model preference versus 50% human preference, driven by training data artifacts rather than semantic content.
Paired Know and Act tests across 16 systems revealed a large gap: agents pass recall tests but fail to reflect preferences in behavior. Comprehension failures during interpretation dominate over retrieval failures, suggesting the bottleneck lies in applying stored information rather than retrieving it.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Measuring Human Preferences in RLHF is a Social Science Problem
- Capturing Individual Human Preferences with Reward Features
- Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
- Beyond Preferences in AI Alignment
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Self-Improving Model Steering
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries