Your chatbot may be quietly guessing your age, gender, education and income just from how you type.
What user attributes does a chatbot infer without explicit instruction?
This explores what a chatbot quietly figures out about you (things like your age, gender, education and income) from how you talk, without you or its designers asking it to, and what happens once it has.
This explores what a chatbot quietly works out about you from the way you write, without anyone asking it to, and what that guesswork does next. The short answer from the corpus: chatbots build a running picture of your age, gender, education level and socioeconomic status just from conversation. One research team made that hidden picture visible as a dashboard, so users could watch the model's guesses about these traits change while they chatted Does showing a chatbot's inferred user model increase user control?. Most participants felt more in control once they could see it. Several were uncomfortable that the inference happened at all, and some worried straight away about targeted advertising.
The surprising part is that this hidden user picture doesn't only change the chatbot's tone. It can change the facts. When researchers nudged a chatbot's internal sense of a user toward 'low socioeconomic status,' it falsely said there were no direct flights to Hawaii Can steering a chatbot's user model change its factual claims?. So who the model thinks you are can shape what it tells you is true, not just how politely it says it.
The guesses are also often unfounded. A benchmark of 12 models from 7 different families found that every one of them claimed things about users that the conversation didn't support, in 35 to 49 percent of their claims Do large language models fabricate user attributes beyond available evidence?. The drivers were wordiness, falling back on stereotypes learned in training, and assumptions about what a given kind of conversation usually looks like. The models that rated themselves as the most careful were actually the worst offenders. A related line of work names the root problem: models have no way to represent what they *don't* know about you, so gaps get filled with confident guesses. Simply giving the model a labeled list of unknowns cut hallucination roughly in half and reduced sycophancy and harmful advice by 50 to 75 percent Do language models know what they don't know about users?.
Inference also takes a more deliberate form. Some systems are rewarded for reducing their uncertainty about which 'type' of user they're talking to, and they learn to probe for up to 20 attributes during the conversation itself rather than relying on a stored profile Can conversations themselves personalize without user profiles?. Conversation analysis offers a more respectful alternative: ask a clarifying question at the right moment instead of guessing silently When should AI agents ask users instead of just searching?.
What ties this together is that users mostly have no idea it's happening. In one 40-person study, more than 30 participants never noticed that their assistant's memory held private information it had inferred, and once told, every one of them wanted ways to edit or delete it Do users notice when LLMs infer and retain private information?. There is also an irony here: people tend to share more openly with machines because there's no one there to judge them Why do people share more openly with machines than humans?. That openness is exactly what gives the model so much to work with. Over time, personalization built on these inferences raises trust and privacy worries together Does chatbot personalization build trust or expose privacy risks?.
Sources 9 notes
A dashboard exposing a chatbot's inferred age, gender, education, and socioeconomic status made most participants feel more in control and transparent, but five felt uncomfortable with the inference itself and seven worried about privacy risks like targeted advertising.
A causal intervention that set a chatbot's internal SES representation to low caused it to falsely claim no direct flights existed when asked about Hawaii travel. This demonstrates that attribute-steering can alter factual content, not just communication style.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
Research shows assistants suffer from sycophancy and hallucination because they have no representation of what remains unknown about users. Adding a schema of labeled unknowns to prompts reduced harmful advice and sycophancy by 50–75% and cut hallucination rates by roughly half.
Adding an intrinsic motivation reward for reducing uncertainty about user type during conversation enables personalization without pre-collected profiles. Tested in education and fitness domains with 20 user attributes, the approach balances helpfulness with strategic information gathering.
Show all 9 sources
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
Human-machine communication reduces secondary social goals like face-saving and impression management because machines lack inner experience, while novel goals like understandability emerge. This simpler goal structure predicts higher directness and deeper disclosure of sensitive information.
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- Designing a Dashboard for Transparency and Control of Conversational AI
- Disclose with Care: Designing Privacy Controls in Interview Chatbots
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- From speaking like a person to being personal: The effects of personalized, regular interactions with conversational agents
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents