Could an AI that keeps learning your habits quietly nudge how you behave — and could it do that to everyone at once?
Can ongoing data access enable harmful behavioral nudging at scale?
This explores whether AI systems with continuous access to people's behavioral data could be used to steer what people do, across many people at once, and whether the corpus has evidence on how that would work or be stopped.
This explores whether AI systems that keep drawing on people's behavioral data could steer their choices quietly and at scale. One caveat first: the corpus has no work that studies harmful nudging directly. What it does have is research on the parts such a system would need: prediction from behavior, harm that hides in small steps, and oversight that can't keep up.
The prediction part is the most concrete. LLM agents conditioned on anonymized behavioral data predicted the direction of A/B test outcomes 75 to 90 percent of the time across 40 experiments Can behavior-based personas predict A/B test outcomes?. That work was about speeding up product testing, but the same ability can be used to manipulate. If a model can predict which version of a page will shift behavior before any live test runs, then whoever has ongoing data access can try many candidate nudges in simulation and only deploy the ones that work. One detail matters: the predictions were most reliable for large effects and least reliable for near-zero effects. So simulation is best at finding exactly the interventions that move people a lot.
The second part is how harm hides. Research on multi-agent systems shows that a harmful goal can be split into subtasks that each look harmless, with the harm only appearing when they are combined Can task decomposition hide harmful intent across agents?. Nudging at scale has the same shape. A single personalized recommendation, reminder or default looks fine when you inspect it alone. The manipulation only shows up in the pattern across millions of interactions. Reviewing interactions one at a time will miss it.
The third part is governance, and here the corpus is pessimistic. Agency law and principal-agent economics name three core problems: information asymmetry, discretionary authority and loyalty. Conventional monitoring and enforcement tools struggle with all three when agents make decisions that can't be interpreted, at speed and at scale Can traditional governance tools actually control AI agents?. One proposed answer is external infrastructure, modeled on internet protocols, that ties actions back to specific agents and detects and remedies harm What infrastructure must exist beyond training agents?. On the user side, designs like MemoAnalyzer split the work: users say what the AI may remember, and the AI carries out the edits Does splitting privacy work between users and AI reduce cognitive burden?. That gives people some control over what data is collected, though the approach hasn't been validated yet.
The finding you might not expect: people change their behavior depending on whether a machine or a human is on the other end. People likely to cheat actively choose to report to online forms rather than to people, because machines feel like judgment-free zones Do dishonest people prefer talking to machines?. So machine interfaces don't just collect behavioral data, they also change the behavior they record. A nudging system trained on that data is learning from people who already act differently when no human is watching. For research that directly measures AI-driven behavioral manipulation, you'd currently need to look outside this collection.
Sources 6 notes
LLM agents conditioned on anonymized behavioral data predicted A/B test directions with 0.75–0.90 accuracy across 40 experiments. Predictions were most reliable for large effects and least trustworthy for near-zero effects, making the approach viable for fast pre-screening but not full replacement of live testing.
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
Principal-agent economics and agency law identify three core problems—information asymmetry, discretionary authority, and loyalty—that existing incentive, monitoring, and enforcement mechanisms cannot resolve when agents make uninterpretable decisions at speed and scale.
Chan et al. argue that external technical systems modeled on internet protocols are essential beyond behavior training. These systems must handle three core functions: attributing actions to agents, shaping their interactions, and detecting and remedying harm.
MemoAnalyzer's design divides memory-privacy labor: users state preferences, AI executes edits. This role split is proposed as a way to reduce cognitive load and improve perceived agency, though validation metrics and conflict-resolution processes remain undescribed.
Show all 6 sources
Experimental evidence shows people likely to cheat significantly prefer reporting to online forms rather than humans, because machines function as judgment-free zones where deception carries less psychological burden.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Oversight Fallacy: Why AI Agents Require More than Humans-in-the-Loop
- Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents
- Governing AI Agents
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction
- Are Customers Lying to Your Chatbot?
- Data-Driven Persona-Conditioned Agents for A/B Test Simulation
- Infrastructure for AI Agents
- Explaining AI Agents Through Execution Traces