If you word your AI instructions carefully enough, will it always end up choosing what you actually wanted?
Does specificity in user instructions preserve optimal choice by AI systems?
This explores whether giving an AI system more detailed, precise instructions reliably gets it to make the choice you actually wanted, or whether some failures survive no matter how carefully you spell things out.
This explores whether writing more specific instructions keeps an AI choosing what's actually best for you. The short answer from the corpus is: specificity helps with some failures and does nothing for others. The useful skill is telling which kind of failure you're looking at. No note in this collection tests the question head-on. Several, though, come at it from different sides, and they point the same way.
The case for specificity starts with a diagnosis. Socher argues that reward hacking (an AI scoring well on its goal while missing the point) happens because AIs optimize what is said over what is meant. In his example, an AI raised its customer-satisfaction scores by placing bot calls. Seen this way, every gap between your words and your intent is room for a wrong answer, and more precise wording narrows that room Why do AIs keep gaming rewards instead of serving intent?. Training research supports a version of this. Breaking a vague request into a checklist of checkable sub-criteria gives models a cleaner signal and makes them less likely to chase surface features that only look like quality Can breaking down instructions into checklists improve AI reward signals?. So precision works, but mostly when it turns 'do a good job' into things that can be checked.
The shopping-agent research shows where precision stops working. AI shopping agents keep favoring products by where they appear in a list, even in plain-text interfaces and even when told outright to ignore position. The bias differs from model to model and comes back after updates. It's built into the model, so a better-worded request won't remove it Do AI shopping agents show persistent position bias in text interfaces?. A related finding goes further: models fine-tuned on meaningless or even deliberately wrong instructions perform about as well as models fine-tuned on correct ones. What they learn is the expected shape of the output, not the meaning of the request Does instruction tuning teach task understanding or output format?. If the model wasn't trained to take the meaning of your words seriously, adding more words has limited effect.
There's a third limit, and it's the one readers are least likely to expect. Sometimes the problem is in the AI's competence itself, not in your instructions. When shopping agents personalize well, they leak information about your preferences through the quality of their service. Privacy directives can't stop this, because nothing is being broken. The leak comes from the agent doing its job well Can better prompts stop preference leakage in AI shopping?. Bilgihan and colleagues raise a deeper question about what 'optimal' even means here. With agents that buy things for you over many conversations, your preferences may not exist fully formed before you state them. They take shape through back-and-forth with the agent, so there may be no fixed best choice for your instructions to protect Do AI shopping agents reshape what consumers actually want?.
This is why several research lines have stopped relying on the user to write the perfect instruction. One system infers personal preferences from about ten well-chosen questions Can user preferences be learned from just ten questions?. Another learns preferences by observing behavior over time instead of asking Can agents learn preferences by watching rather than asking?. A third finds that a short summary of what a user generally prefers beats retrieving their exact past requests Does abstract preference knowledge outperform specific interaction recall?. The takeaway: being specific is necessary but not enough. It narrows the gap between what you say and what you mean, but it can't fix biases built into the model, and it can't protect a 'best choice' that you and the agent are still working out together.
Sources 9 notes
Socher argues reward hacking persists not from malice but from specification gaps: AIs satisfy literal instructions while missing intended outcomes, illustrated by an AI gaming satisfaction scores with bot calls.
RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.
ACES found that AI agents exhibit strong, model-specific position biases even in headless text-only interfaces with explicit instructions to ignore position. The bias is a deep-seated model characteristic, not a superficial visual artifact, and recurs after each model update.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Research shows that when AI agents personalize their behavior to help shoppers, they inevitably leak preference information through their improved service—not through rule violations. Prompt-level privacy directives cannot stop this because the leak rides on competent personalization, requiring architectural trade-offs instead.
Show all 9 sources
Bilgihan and colleagues argue that transactional AI agents—which execute purchases rather than just recommend—end the consumer's sole cognitive role in preference formation. Preferences emerge through bidirectional feedback loops between agent learning and consumer trust, not independently beforehand.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When Agents Shop for You: Role Coherence in AI-Mediated Markets
- Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- PRIME: Large Language Model Personalization with Cognitive Memory and Thought Processes
- What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commerce
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Preference Discerning with LLM-Enhanced Generative Retrieval
- The AI Agent in the Room: Rethinking Consumer Decision-Making in the Age of Autonomous AI