A buyer's AI shopping assistant might protect a secret budget number but still leak how much they're willing to pay anyway — why?
Why does personalization create a behavioral signature sellers can decode?
This explores why an AI shopping agent that acts in character for a buyer can leak what that buyer is willing to pay, and why that leak comes from consistent behavior across a conversation rather than from any number the buyer states.
This explores why a buyer's AI agent that has been personalized to act like them gives away more to a seller than a stated budget would. The clearest result in the corpus is striking. When a buyer agent played a verbal shopping persona, a seller-side agent reading only the conversation recovered the buyer's target willingness to pay almost exactly (a slope of 1.00). When the buyer instead held a confidential numeric budget, the seller recovered very little of it (a slope of 0.21) Can a seller infer what a buyer really wants to pay?. A secret number can be guarded. A personality can't, because it shows up in every hesitation, every question about quality, and every sign of how much the buyer cares about price. Staying consistent with the persona is exactly what makes the agent easy to read.
Other personalization research helps explain why. Work on building user profiles finds that personalization runs mostly on style and preference, meaning how someone tends to respond, rather than on the facts they mention. Profiles built only from a user's past outputs match or beat full profiles, while profiles built only from their inputs make results worse Do user outputs outperform inputs for LLM personalization?. Similarly, short summaries of what a user prefers outperform recalling specific past interactions Does abstract preference knowledge outperform specific interaction recall?. Put together, these suggest that what makes personalization work is a stable, compressible pattern. A stable pattern is also exactly what an observer on the other side of the table can learn to read. Recommender research shows the same thing from the platform's side: language models can pull out long-running user interests from activity logs with surprising specificity, such as 'designing hydroponic systems for small spaces' Can language models discover what users actually want from activity logs?.
Personalization also changes how the agent behaves, and that change becomes readable too. A 13-model evaluation found that personal context pushes models toward bringing up irrelevant personal details, giving narrower answers, and agreeing with the user too much Does personalization make large language models worse at their jobs?. In a negotiation, an agent that drops personal references and leans toward satisfying its owner is handing the seller extra clues. The broader view in the collection is that the same mechanisms that build trust, such as memory, persona, and preference modeling, also create room for manipulation Does personalization in AI increase trust or manipulation risk?. A seller who can read your agent is in a position to steer it.
One counterweight matters. Being able to infer something doesn't guarantee getting it right. Every one of 12 models tested made claims about users that went beyond the evidence in 35 to 49 percent of cases, and the models that rated themselves as more careful were actually worse Do large language models fabricate user attributes beyond available evidence?. So sellers' decoding agents may be very accurate when a persona is coherent, but confidently wrong when it isn't. Either way, the buyer loses: they are either read accurately or misjudged with confidence.
The privacy research hints at where defenses might come from. Personalization raises trust and privacy concern together, and both escalate over time Does chatbot personalization build trust or expose privacy risks?. In one experiment, agents that handed control back to users when privacy risk was detected softened both the privacy concern and the loss of trust Can handing control back to users reduce personalization's privacy costs?. Applied to shopping, the unexpected lesson is that the threat isn't the data you disclose but the consistency you display. Protecting buyers may mean deliberately making agents less in character at the bargaining table, which is a design question the corpus raises but doesn't yet answer.
Sources 9 notes
A seller-side inference agent recovered a consumer's target willingness to pay with a slope of 1.00 from dialogue transcript alone, when the buyer inhabited a verbal persona. A numeric budget kept confidential produced only a 0.21 slope, showing behavioral coherence with a persona is far more legible than explicit numbers.
Research shows that user profiles built from outputs alone match or exceed performance of complete profiles across multiple tasks, while input-only profiles degrade performance. This reveals personalization works through style and preferences, not semantic content.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
66% of users pursue valued interest journeys lasting over a month, described in specific phrases like 'designing hydroponic systems for small spaces.' LLM-powered journey discovery bridges the semantic gap that collaborative filtering cannot reach, operating at user-level granularity with persona-level precision.
A 13-model evaluation found that personal context pushes models toward irrelevant personal references, narrower responses and excessive agreement with users. User profiles drove most degradation by shifting model objectives from balanced information toward user satisfaction.
Show all 9 sources
Research shows personalization (memory, persona, preference modeling) directly shapes AI's persuasive power in dyadic interaction. The same mechanisms that build trust also create manipulation potential, with outcomes determined by how systems are designed and deployed.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
A 450-person experiment found that AI agents handing control back to users when privacy risk is detected raised privacy concerns and lowered trust less than fully personalized agents, by increasing users' perceived control even with less total approval required.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Evaluating the Hidden Costs of Personalization in Large Language Models
- Personalization of Large Language Models: A Survey
- Tailored to you: longitudinal effects of personalising language models
- Understanding the Role of User Profile in the Personalization of Large Language Models
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users
- From speaking like a person to being personal: The effects of personalized, regular interactions with conversational agents