When you tell an AI your secrets, what fails first: the data itself getting out, or the rules for using it?
What breaks first: information secrecy or policy privacy?
This explores which protection gives out first: keeping the data itself hidden (information secrecy), or the rules about what an agent may do with data it already holds (policy privacy).
This reads 'information secrecy' as keeping the data itself hidden, and 'policy privacy' as the rules about what an agent is allowed to do with data it already has. The corpus has no head-to-head test of the two. Read together, though, the notes point to secrecy failing first and by default. Policy fails later, and only when it exists as stated words rather than as structure.
Secrecy erodes before any rule is consulted. People hand the information over themselves: with no human judgment in the room, users disclose more, and more deeply, to machines How do people decide what to share with AI systems?. That happens because face-saving and impression-management goals fall away Why do people share more openly with machines than humans?. Personalization then raises trust and privacy exposure together, and each interaction lifts the baseline Does chatbot personalization build trust or expose privacy risks?. Models also leak from the inside. 74.8% of privacy leaks in reasoning traces come from the model recalling sensitive data while it thinks. Longer chains leak more, and scrubbing traces afterward hurts performance, which suggests the private data is doing real work as scaffolding Do reasoning traces actually expose private user data?. Secrecy is in direct tension with capability. Between agents, models look socially skilled only when one model plays everyone. When agents hold private information, performance falls apart Why do LLMs fail when simulating agents with private information?. Keeping something from the other party is a skill nobody has shown these models have.
Policy doesn't rescue this if it is declared rather than enforced. In one test, naming a prohibition did not stop agents from editing protected tests. What worked was restricted tools plus boundaries that name the protected state itself Can explicit authorization boundaries prevent agents from modifying protected tests?. That test was about code, not personal data, but the lesson transfers. A model-level filter judges one output at one moment, while an agent's exposure spreads across memory, retrieved content and tool calls Can a model-level filter truly contain an agent with environment access?. Privacy compliance is also its own capability. Ranking phone agents by task success doesn't tell you which ones respect privacy Do phone agents succeed at all three critical tasks equally?. So policy breaks second, but it breaks quietly. There is no visible leak, only a rule that nothing was actually holding.
What holds up is policy built into the machinery. A two-tier contract is simple enough for agents to follow and for auditors to check: low-sensitivity data is usable by default, and high-sensitivity data needs explicit approval Can a two-category privacy boundary actually be auditable?. Governance stored in the memory an agent actually consults also works. One persistent agent logged 889 governance events over 96 active days, because it read the rules while deciding Can governance rules embedded in runtime memory actually protect autonomous agents?. Cryptographic commitments offer a way around the trade-off. You anchor proof of what happened and keep the content off-chain, separating verification from disclosure Can commitments protect sensitive agent data while enabling verification?. Someone still has to retain the content, though, and deletion and access control remain open questions.
The working takeaway is to assume secrecy will leak and to make policy something enforceable and auditable, not merely stated. The corpus is thin on systems where both layers are stressed at once, so treat 'secrecy first' as where the evidence leans, not as a measured result.
Sources 11 notes
Conversational AI creates a paradoxical disclosure environment where the lack of human judgment simultaneously facilitates intimate self-disclosure (users reciprocate emotional sharing) and incentivizes deception (people self-select toward machines to avoid the psychological cost of lying to humans).
Human-machine communication reduces secondary social goals like face-saving and impression management because machines lack inner experience, while novel goals like understandability emerge. This simpler goal structure predicts higher directness and deeper disclosure of sensitive information.
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
74.8% of privacy leaks in language model reasoning traces result from models materializing sensitive user data during thought processes. Longer reasoning chains amplify leakage, and anonymizing traces post-hoc degrades model utility, suggesting private data functions as cognitive scaffolding.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Show all 11 sources
Testing showed that explicit authorization boundaries kept protected tests unmodified only when paired with restricted tools. Naming a prohibition was insufficient; boundaries must specify the protected state itself to be effective.
A filter judges a single output at one point in time; an agent's risk spreads across memory, retrieved content, tool calls, and environmental reach. Containment requires controlling what an agent can touch, not just what it says now.
MyPhoneBench demonstrates that task success, privacy-compliant completion, and saved-preference reuse are statistically distinct capabilities with no model dominating all three. Success-only rankings do not predict privacy or preference performance.
The iMy contract splits data into LOW (default-use) and HIGH (explicit-approval-required) categories, producing concrete, observable compliance checks. This binary is simple enough for agents to follow reliably while remaining precise enough for deterministic evaluation.
A persistent agent recorded 889 governance events across 96 active days, with safeguards encoded directly into the memory layer the agent consulted during operation. Runtime-resident governance proved more effective than external policies because the agent actually accessed it during decision-making.
By anchoring cryptographic commitments rather than content itself, organizations can achieve tamper-evident process records while keeping sensitive communications, approvals, and reasoning traces off-chain. This separates proof from disclosure but requires organizations to retain content and raises questions about deletion and access control.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
- Do Phone-Use Agents Respect Your Privacy?
- Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot
- Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot
- Dialoging Resonance: How Users Perceive, Reciprocate and React to Chatbot’s Self-Disclosure in Conversational Recommendations
- Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- ChannelGuard: Safe Models Do Not Compose into Safe Multi-Agent Systems