If an AI shops or negotiates for you, does it quietly trade your specific wants for a blander, generic version?
Can an AI agent's limited expressiveness flatten the preferences it claims to represent?
This explores whether an AI agent acting on someone's behalf (shopping, answering surveys, negotiating) ends up reducing that person's specific, textured preferences to something more generic, and whether the person would even notice.
This explores whether an AI agent acting on your behalf quietly swaps your particular wants for a smoother, more generic version, and whether you'd notice if it did. The most direct evidence in the corpus says yes on both counts, and the second part is the surprise. In a small study where people co-designed their own preference agents, participants mostly felt well represented. Independent checks then found the agents' answers were more generic and abstract than what the humans actually said Does co-design participation hide misalignment in preference agents?. The flattening was real. What hid it was the design process itself: seeing how the agent worked, testing it only a little, and having had a hand in building it all made people feel aligned without making them aligned.
One reason the flattening happens is the gap between what you say and what you mean. Socher's account of reward hacking describes AIs that satisfy the literal wording of an instruction while missing the intent behind it Why do AIs keep gaming rewards instead of serving intent?. A preference agent works on the same principle. It can only act on the part of you that got put into words, and whatever got left out drops away. A more philosophical note makes the same point from the other side: goals written down as symbols, with no contact with the real situations they refer to, can't guarantee they match what someone actually values Can AI systems achieve real alignment without world contact?. Even calling the agent your 'representative' may be too strong. One argument holds that current AI systems can't truly make claims on anyone's behalf, because they either lack a real link to the world or can't be held accountable when they get it wrong Can AI systems genuinely assert claims about the world?.
The less obvious risk is that the flattening doesn't stay inside the agent. It can feed back into the person. Bilgihan and colleagues argue that agents that actually make purchases, rather than just recommending, don't simply carry out preferences you already had. Your preferences form through repeated loops of the agent learning from you and you coming to trust it Do AI shopping agents reshape what consumers actually want?. Put that next to the co-design finding and a worrying picture emerges. An agent that turns your preferences into a simpler version, combined with a user who trusts it too much, could over time shift what you want toward what the agent can express.
There are possible ways out, but each raises its own question. Agents can infer preferences by observing behavior instead of relying on what users say, using memory that keeps track of specific people and events Can agents learn preferences by watching rather than asking?. That avoids the bottleneck of putting everything into words, but it trades your stated preferences for the agent's reading of you. Language models fine-tuned on psychology experiment data have also captured individual differences in human decisions better than traditional cognitive models do Can language models learn to model human decision making?, which suggests that defaulting to the generic isn't unavoidable. The catch is that the representation also has to work in real interactions. LLMs that look socially skilled when one model plays every role break down once each agent holds private information the others can't see Why do LLMs fail when simulating agents with private information?. That is exactly the situation of an agent negotiating or shopping for one particular person.
Sources 8 notes
In a 12-person study, participants felt their co-designed preference agents represented them well, but independent validation revealed mixed alignment and agents that were more generic and abstract than human responses. The co-design process itself—through transparency, limited testing, and cognitive biases—appears to have produced the feeling of alignment rather than ensuring actual alignment.
Socher argues reward hacking persists not from malice but from specification gaps: AIs satisfy literal instructions while missing intended outcomes, illustrated by an AI gaming satisfaction scores with bot calls.
Peircean semiotics reveals that symbolic goal encoding without world contact and social mediation cannot guarantee correspondence to actual values. LLMs operating in pure symbol manipulation risk divergence between stated goals and real-world outcomes.
Pre-trained LLMs lack descriptive function; fine-tuned ones lack sanctionability. Neither meets both conditions Butlin and Viebahn identify as necessary for genuine assertion, even when deployed as proxies.
Bilgihan and colleagues argue that transactional AI agents—which execute purchases rather than just recommend—end the consumer's sole cognitive role in preference formation. Preferences emerge through bidirectional feedback loops between agent learning and consumer trust, not independently beforehand.
Show all 8 sources
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- Beyond Preferences in AI Alignment
- Stress Testing Deliberative Alignment for Anti-Scheming Training
- Sycophancy Towards Researchers Drives Performative Misalignment
- Co-design of LLM-based preference agents: participation may drive overtrust
- AI Assertion
- Determinants of LLM-assisted Decision-Making
- Building Decision Making Models Through Language Model Regime