If an AI agent only promises not to share your name or address, is that enough — or does it still leak what it infers about you?

This explores whether a privacy limit built around categories of factual data, such as 'share this, never share that,' actually covers the ways AI systems expose people. The corpus has no paper that tests 'disclosure budgets' under that name. It does have enough nearby material to suggest an answer: factual-category limits are useful and checkable, but much of the real exposure happens in places those limits don't look.

The case for category-based limits is real. The iMy privacy contract sorts user data into just two buckets: LOW, which the agent can use by default, and HIGH, which needs explicit approval. That binary is simple enough for agents to follow reliably and precise enough to audit automatically Can a two-category privacy boundary actually be auditable?. That's the strength of counting factual data types: you can tell whether the rule was broken. The weakness is that the rule only governs data that already exists as a labeled fact.

The bigger problem is that models produce facts about you that you never gave them. When MirageBench tested 12 LLMs, every one of them made claims about users that went beyond the evidence in 35–49% of cases. Those claims drew on pretraining stereotypes and genre expectations, and the models that rated themselves as over-inferring least actually did it most Do large language models fabricate user attributes beyond available evidence?. A budget that tracks 'did the user share their health status?' misses the model concluding it anyway. Users don't catch this either. In one 40-person study, more than 30 participants never noticed that their assistant's memory held private information it had inferred. Once they were shown it, every participant wanted to edit or delete it Do users notice when LLMs infer and retain private information?. Inferred data is the category nobody is budgeting for, and nobody can see it.

There's also a leak that happens before any output. Reasoning traces expose private data mostly because the model restates sensitive details while thinking: that accounts for 74.8% of the leaks studied. Longer reasoning leaks more, and scrubbing the traces afterward hurts performance. That suggests the private data is doing real work in the model's reasoning Do reasoning traces actually expose private user data?. A budget enforced at the moment of sharing never sees this channel. One structural alternative separates proof from content. Agents publish cryptographic commitments, which let others verify that a record wasn't tampered with without seeing what it says, and keep the sensitive reasoning private Can commitments protect sensitive agent data while enabling verification?.

A useful idea comes from an unrelated area. Work on value bias in models finds that 'how much the model leaked' and 'whether it admitted leaking' are separate measurements that can point in opposite directions Do models that leak values also disclose those leaks?. The honesty standard that work proposes is disclosure: bias that's acknowledged can be priced in, and hidden bias can't Should models disclose their value biases when neutral answers are impossible?. Applied to privacy, a budget that only counts factual items shared is a single score, and it hides how the exposure happened. A stronger design would also show users what the system inferred about them and where their data showed up along the way, not just which labeled fields went out. The corpus doesn't yet test such a design directly, so that part is an open question.