If an AI learns you like romantic restaurants, what extra labels would help it guess your taste in films or gifts?
What explicit concept annotations would improve cross-concept preference reasoning?
This explores what extra labels, beyond "response A beat response B", would let a model carry a preference learned in one area (say, romantic restaurants) over to another (romantic films, gifts).
This explores what extra labels, beyond "response A beat response B", would let a model carry a preference learned in one area over to another. The corpus has no paper that tests concept annotations for this directly. Several pieces do point the same way: annotate *what kind of signal* it is, *which concept* drove it, and *how it connects* to other judgments.
The first annotation is the kind of signal. Do all annotation responses measure the same underlying thing? finds that annotations mix genuine preferences, non-attitudes (the person had no view), and constructed preferences (invented on the spot from how the question was framed). You can tell them apart by whether the answer holds up across different ways of asking. My inference is that only genuine preferences should transfer across concepts, and a constructed one mostly encodes the wording of the survey. A label for the signal type keeps cross-concept reasoning from learning from noise. Compare Can crowdsourced votes reliably rank language models?. Bare pairwise votes rank models credibly because the questions are diverse and the crowd matches expert raters. A vote records who won, though, not which concept decided it.
The second annotation is the concept itself, in plain words. Can language models bridge the gap between critique and preference? shows a language model rewriting "doesn't look good for a date" as "prefer more romantic". That rewrite is a reusable concept label: "romantic" can be matched against other kinds of items, while the original complaint stays tied to one item. Can models learn behavioral principles without preference labels? goes the other direction. It links responses to written principles by maximizing the mutual information between them, with no preference labels at all, and a weaker model's principles were enough to align a stronger one. The written principle works as the annotation. Does abstract preference knowledge outperform specific interaction recall? points the same way: abstract preference summaries beat retrieving specific past interactions for personalization. Transfer seems to live at the level of the abstraction, not the individual episode.
The third annotation is how judgments connect. What alignment data structure best trains reasoning generalists? organizes alignment data as a tree per instruction. The tree holds diverse reasoning chains, critiques and pairwise comparisons together, so related judgments sit next to each other. Can agents learn preferences by watching rather than asking? attaches preferences to entities. It keeps events ("she ordered the candlelit place") apart from lasting knowledge ("she likes intimate settings"), so a preference is bound to a person and a concept, not to a single moment.
The open question is whether explicit labels are needed at all. Can language models learn to model human decision making? shows models finetuned on varied human decision data capture individual differences in their embeddings and transfer across tasks without task-specific design. Enough varied data may teach the concept structure implicitly. Nothing here compares that against hand-labeled concepts, so that comparison is the gap worth testing.
Sources 8 notes
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Chatbot Arena's 240K+ crowdsourced preference votes produce credible model rankings because the underlying questions are diverse and discriminating, and crowd judgments correlate with expert raters—validating human preference as a scalable evaluation signal.
Few-shot LLM prompting can convert natural negative feedback like "doesn't look good for a date" into positive preferences like "prefer more romantic," enabling retrieval systems to find better-matching recommendations without fine-tuning.
SAMI finetunes language models to increase mutual information between constitutions and responses without preference labels or demonstrations. A mistral-7b trained this way outperformed base and instruction-tuned baselines, and surprisingly, a weaker model could write principles to align a stronger one.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
Show all 8 sources
Eurus achieved state-of-the-art open-model reasoning by training on ULTRAINTERACT, an alignment dataset structured as preference trees per instruction. The tree format unified diverse planning strategies, interaction-and-critique trajectories, and pairwise data for both SFT and preference learning.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Preference Discerning with LLM-Enhanced Generative Retrieval
- PRIME: Large Language Model Personalization with Cognitive Memory and Thought Processes
- Measuring Human Preferences in RLHF is a Social Science Problem
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- PersonaAgent: When Large Language Model Agents Meet Personalization at Test Time
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
- Beyond Preferences in AI Alignment