Your AI remembers what you like but still ignores it — how do you test whether it understood?
How should comprehension failures during preference application be measured and operationalized?
This explores how to test whether an AI agent that has stored a user's preference actually understands how to apply it, and how to tell that failure apart from simply not finding the preference.
This is about catching an agent that knows what you like but doesn't act on it, and separating "couldn't find it" from "didn't understand how to use it." The corpus has one clear measurement design for this. Test the same preference twice: a Know test (can the agent recall it?) and an Act test (does its behavior reflect it?). Across 16 systems, agents passed recall but failed to act, and the failures were mostly comprehension during interpretation, not retrieval (Why do LLM agents remember preferences but not act on them?). So the operational definition is the gap between the two scores, not a single accuracy number.
A gap tells you something failed after recall, but not what. The corpus doesn't give a standard scoring protocol for that step, so the next moves are inferences from neighboring results. One is to change what the agent is handed and see whether behavior recovers. In bandit tasks, models that couldn't use raw interaction history did fine once it was summarized externally (Why do LLMs struggle with exploration in simple decision tasks?). Abstracted preference summaries also beat retrieved past interactions for personalization (Does abstract preference knowledge outperform specific interaction recall?). If a clean, pre-digested preference fixes the Act score while the raw one doesn't, the bottleneck is interpretation rather than memory.
A second signal is what the agent does when it's unsure. Comprehension failures are often silent. Models produce 77.5% fewer grounding acts (clarifying questions, understanding checks) than humans, and preference optimization makes this worse (Does preference optimization damage conversational grounding in large language models?, Does preference optimization harm conversational understanding?). An agent trained to sound confident will apply a preference wrongly without flagging any doubt. Counting how often it checks its reading of a preference is therefore a behavioral trace of comprehension that outcome-only scoring misses.
Third, the test preference itself has to be trustworthy. Annotation research shows that responses mix genuine preferences, non-attitudes, and constructed preferences, and that you can tell them apart by whether they stay consistent across measurement conditions (Do all annotation responses measure the same underlying thing?). The same logic applies to Act tests. Probe one preference under varied phrasings and contexts, so that inconsistent application can be pinned on the agent rather than on a shaky preference. The form of the preference matters too. Negative feedback like "doesn't look good for a date" has to be converted into a positive one like "prefer more romantic" before it's usable (Can language models bridge the gap between critique and preference?). That conversion is a natural, separately testable comprehension step.
So the corpus supports a layered approach: a Know/Act gap to detect the problem, input interventions to locate it, grounding behavior to catch silent failures, and consistency checks to validate the test itself. It doesn't yet offer a validated rubric for grading how an agent misread a preference. That part is open.
Sources 7 notes
Paired Know and Act tests across 16 systems revealed a large gap: agents pass recall tests but fail to reflect preferences in behavior. Comprehension failures during interpretation dominate over retrieval failures, suggesting the bottleneck lies in applying stored information rather than retrieving it.
Across multi-armed bandit environments, only GPT-4 with explicit exploratory hints, external history summarization, and chain-of-thought reasoning achieves satisfactory exploration. Without external summarization, models cannot reliably track and aggregate unstructured interaction history to guide exploratory decisions.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
Research shows LLMs generate 77.5% fewer grounding acts than humans, and RLHF preference optimization actively worsens this gap. The optimization target—fluent, confident responses—directly undermines the communicative work of establishing shared understanding.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
Show all 7 sources
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Few-shot LLM prompting can convert natural negative feedback like "doesn't look good for a date" into positive preferences like "prefer more romantic," enabling retrieval systems to find better-matching recommendations without fine-tuning.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Grounding Gaps in Language Model Generations
- Preference Discerning with LLM-Enhanced Generative Retrieval
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- When Large Language Models contradict humans? Large Language Models’ Sycophantic Behaviour
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
- Can large language models explore in-context?