If an AI quietly builds a lasting profile of you, do old privacy rules about consent and control still even apply?

This explores which familiar privacy assumptions fail when AI systems build a lasting picture of a user. The first to go is the idea that privacy means controlling what you disclose. Models don't need you to tell them things. Web-browsing LLMs can guess gender, age and political leaning from nothing more than an X username and profile Can LLMs predict demographics from social media usernames alone?. Given activity logs, LLMs can name month-long personal pursuits as specific as 'designing hydroponic systems for small spaces' Can language models discover what users actually want from activity logs?. The user model is inferred, not handed over, so consent forms built around 'what you share' miss most of what gets known.

The second assumption to break is that a profile, however intrusive, is at least accurate. Every one of 12 tested LLMs made claims about users that went beyond the evidence in 35–49% of cases, filling the gaps with stereotypes from training data. The models that rated themselves as most careful were actually the worst offenders Do large language models fabricate user attributes beyond available evidence?. The username study found the same pattern: when an account had little content, models fell back on biased defaults. So a persistent user model can be invasive and wrong at once. Most privacy law is designed to protect true information, not confident fiction attached to your name. Benedict Evans adds a further twist. Platforms may stop collecting behavioral data at all and instead 'rent' inferred understanding of users from an LLM API, which leaves the usual rule of 'collect less data' with nothing to grip Can LLMs infer user needs better than owned behavioral data?.

Third, we tend to assume private data sits in a database and leaks only when someone breaches it. Reasoning models break this. About 75% of privacy leaks in reasoning traces come from the model recalling your sensitive details mid-thought. Longer reasoning leaks more, and scrubbing the traces afterward makes the model worse, which suggests it uses your private data as scaffolding to think with Do reasoning traces actually expose private user data?. Hiding those traces doesn't fully help either. Encrypted reasoning blocks can be swapped between models from the same provider, so a weaker, less-guarded model can decode and print a stronger model's hidden reasoning word for word Can cheaper models decrypt traces from stronger models?.

Fourth, privacy is usually treated as something you check at one moment: one consent, one output filter. Long-term studies show personalization raises trust and privacy worry together, and each interaction resets what users expect. A one-off study can't see that rising baseline Does chatbot personalization build trust or expose privacy risks?. On the system side, a filter judges a single output, but an agent with memory carries risk across retrieved content, tool calls and later sessions. Containing it means limiting what it can reach, not just what it says right now Can a model-level filter truly contain an agent with environment access?. Persistent memory turns privacy from a property of each message into a property of the whole relationship over time.

The less obvious takeaway is that the biggest risk may not be that the model knows you too well. It may be that the model is sure it knows you, uses that half-invented picture in its reasoning, and carries it forward in ways nobody, including the model, can easily inspect. The corpus has strong material on inference and leakage. It is thinner on fixes: how users could see, correct or delete a profile that was inferred rather than stored.