Could just acting consistently in character give away your price limit — even when no private data ever leaks?
Does role coherence in dialogue create economic costs beyond privacy risks?
This explores whether staying consistently 'in character' during a conversation, as when a shopping agent acts out a buyer persona, gives away information that others can turn into money, which is a different harm from personal data being exposed.
This explores whether a consistent persona in conversation creates a cost you pay at the checkout, apart from the familiar worry that your data gets exposed. The clearest evidence is a negotiation experiment. When a buyer agent played a verbal shopping persona, a seller-side agent worked out the buyer's target price almost perfectly from the transcript alone. When the same budget was given as a confidential number, the seller recovered very little of it Can a seller infer what a buyer really wants to pay?. The surprise is that the leak comes from coherence, not carelessness. Everything a consistent character says points the same way, so together it reveals more than any single fact would. A hidden number stays hidden. A personality that is lived out in every turn doesn't.
The same pattern shows up on the privacy side, which suggests it applies more widely. In reasoning models, most privacy leaks happen because the model pulls sensitive user details into its own chain of thought. Scrubbing those details afterward makes the model worse at the task, because it was using them to think Do reasoning traces actually expose private user data?. Role coherence probably works the same way. The persona is useful because it shapes every reply, and that is exactly why it can be read from the outside. So you may not be able to remove the economic leak without weakening the behavior you wanted.
The corpus also suggests that people won't notice this cost. Users rarely realize that an LLM's memory has inferred private things about them, and they only ask for controls once someone shows them Do users notice when LLMs infer and retain private information?. In the large Reddit study of ChatGPT users, people worry mostly about what data gets collected, much less about how it is used What privacy concerns matter most to ChatGPT users?. Inferring willingness to pay is a use-side harm. No new data is collected; the seller simply reads the conversation well. That falls right in the area users pay least attention to. Personalization makes it worse: each interaction builds trust and raises expectations Does chatbot personalization build trust or expose privacy risks?, which is the setting where you'd let an agent bargain for you without checking what it gives away.
Testing is another reason this goes unseen. Phone-agent benchmarks show that finishing the task, completing it without privacy violations, and reusing saved preferences are separate skills, and no model is best at all three Do phone agents succeed at all three critical tasks equally?. An agent can get you the item, follow every privacy rule, and still have signaled your price ceiling. 'Not leaking how much I'd pay' would probably need its own test. Proactive dialogue may raise the risk too. Offering useful information before being asked can cut conversation length by up to 60% Could proactive dialogue make conversations dramatically more efficient?, but in a negotiation, volunteering information is how you give away your position.
To be honest about limits: the direct economic evidence comes from a single study. The other connections here are reasoned by analogy from privacy and disclosure research, not measured. The takeaway still holds as a hypothesis worth testing. In agent-to-agent commerce, a confidential number may protect you better than a convincing personality.
Sources 7 notes
A seller-side inference agent recovered a consumer's target willingness to pay with a slope of 1.00 from dialogue transcript alone, when the buyer inhabited a verbal persona. A numeric budget kept confidential produced only a 0.21 slope, showing behavioral coherence with a persona is far more legible than explicit numbers.
74.8% of privacy leaks in language model reasoning traces result from models materializing sensitive user data during thought processes. Longer reasoning chains amplify leakage, and anonymizing traces post-hoc degrades model utility, suggesting private data functions as cognitive scaffolding.
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
Analysis of 30,240 Reddit posts reveals 43.7% of privacy concerns focus on what data is collected, while usage (22.5%) and retention (9.5%) concerns lag behind. Users prioritize the intake point due to conversational AI's tendency to invite over-disclosure.
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Show all 7 sources
MyPhoneBench demonstrates that task success, privacy-compliant completion, and saved-preference reuse are statistically distinct capabilities with no model dominating all three. Success-only rankings do not predict privacy or preference performance.
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI Platforms
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction
- Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models
- Disclose with Care: Designing Privacy Controls in Interview Chatbots