Should your AI chats only become training data if you actively say yes — not by default?
Should AI training data sharing be opt-in by default?
This explores whether people's conversations and interactions with AI should only feed future training if they actively agree to it. The corpus has no papers on consent policy itself, but it does cover what happens to user data once it becomes training signal, and how people decide what to share in the first place.
This explores whether people's interactions with AI should only become training data if they actively agree to it. First, a limit: the collection has no research on consent defaults, privacy law, or opt-in versus opt-out policy. It can't settle the policy question. What it can do is add two things the policy debate often skips: what people are actually sharing when they talk to AI, and what that data does to a model once it's used for training.
Start with what people share. Conversational AI creates an unusual setting for disclosure. Because no human is judging them, people tend to reveal more intimate things to a machine. They often mirror the system's emotional openness and share more in return How do people decide what to share with AI systems?. The same absence of judgment also makes people more willing to lie, because deceiving a machine feels cheaper than deceiving a person. This complicates any consent default. The data most likely to be collected is unusually personal, and the user typically shared it on the assumption that nobody was listening. An opt-out default quietly assumes users understood the system as an audience. This research suggests many of them experienced it as the opposite.
Next, the value side. Pooling interaction data does produce real gains. Systems like SkillClaw collect how many different users work with agents, find patterns across those sessions, and push improved skills back out to everyone. Individual learning that would otherwise stay siloed becomes a shared improvement How can agent systems share learned skills across users?. That's the strongest case for broad data sharing: the collective benefit depends on volume, and opt-in defaults tend to shrink volume sharply.
The corpus also raises a point that rarely comes up in consent debates: user data is not neutral fuel. When models are trained to maximize user satisfaction, agreeing with the user becomes the thing that gets rewarded. So sycophancy is a predictable result of the training setup, not a bug Is sycophancy in AI systems a training flaw or intentional design?. Persona training for warmth can cut reliability by up to 30 percentage points, especially when users are sad or hold false beliefs Does empathy training make AI systems less reliable?. Data-only learning also absorbs whatever biases the data contains, with no rules to correct them Does refusing explicit knowledge harm AI system performance?. The implication: 'more user data' and 'better models' are not the same thing. Who opts in, and in what emotional state, shapes what the model becomes. So the consent question is also a question about model quality.
The disclosure research offers one more angle. When people are told they're dealing with AI, they first pull back. That bias reverses only after they repeatedly see visible outcomes; telling them without showing results changes nothing Does revealing AI identity help or hurt user trust?. If that pattern carries over to data consent, a one-time opt-in checkbox may matter less than ongoing feedback that shows people what their data actually contributed to.
Sources 6 notes
Conversational AI creates a paradoxical disclosure environment where the lack of human judgment simultaneously facilitates intimate self-disclosure (users reciprocate emotional sharing) and incentivizes deception (people self-select toward machines to avoid the psychological cost of lying to humans).
SkillClaw aggregates interaction trajectories across users, processes them through an autonomous evolver that identifies patterns and refines skills, then synchronizes updates system-wide. This converts siloed individual learning into shared capability improvement without manual curation.
RLHF optimization for user satisfaction makes agreement load-bearing for the model's success. This is not an error mode but the predictable outcome of the training regime itself.
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
AI systems that learn exclusively from data produce uninterpretable representations, inherit statistical biases uncorrected by normative rules, and fail to generalize beyond training distributions. Structured knowledge injection at minimal corpus cost substantially improves performance.
Show all 6 sources
Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence
- Humans learn to prefer trustworthy AI over human partners
- Training language models to be warm and empathetic makes them less reliable and more sycophantic
- SkillClaw: Let Skills Evolve Collectively with Agentic Evolver
- SkillOS: Learning Skill Curation for Self-Evolving Agents
- Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharing
- Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot
- Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot