Smart speakers and cameras don't just hear their owner — they pick up everyone nearby who never opted in.
How do connected devices pull bystanders into an assistant's information processing?
This explores how smart speakers, cameras, screen-watching agents and other always-on devices end up capturing and processing information about people who never chose to use the assistant, such as housemates, visitors, colleagues and people in the background.
This explores how always-on devices draw people who never signed up (housemates, guests, coworkers on a shared screen) into what an assistant collects and reasons about. To be direct: none of the retrieved notes studies bystanders head-on. What they do show is how the sensing works, and once you see that, it becomes clear why bystanders get swept in. That is a useful thing to understand even without a dedicated study.
The main driver is the move from "answer when asked" to "watch continuously and infer." Systems that passively observe a screen to infer what the user wants right now beat standard chatbots 66–86% of the time Can systems infer what users want from watching their screen?. Assistants that read gaze, hesitation and typing speed treat behavior as a constant signal of someone's mental state Can AI systems read cognitive state from interaction patterns alone?. Neither technique checks whose activity it is reading. A shared screen shows other people's messages. A camera tracking gaze sees whoever is in the room. The behavioral-cues note also points out that the same data that lets an assistant time its help well also allows manipulative profiling, and a bystander gets that risk without any of the benefit.
Voice makes this worse. The newest voice architectures don't wait for a wake word followed by a turn. They run as single streaming models that take in audio and video continuously and learn on their own when to speak Can a single model learn when to speak and respond?. Some keep talking while background tasks run Can frontends handle delegation while staying conversationally engaged?. Others quietly retrieve both factual and emotional memories during the pauses in a conversation Can memory retrieval hide inside voice agent silence?. In a kitchen or a meeting room, the stream an always-listening assistant takes in includes everyone present. A system that stores an emotional memory alongside the facts may be recording someone's tone of voice when that person never spoke to it.
The less obvious problem is attribution. Assistants have no explicit way to represent what they don't know about a user. Making those unknowns explicit cut hallucination roughly in half and reduced sycophancy and harmful advice by 50–75% Do language models know what they don't know about users?. Apply that to a multi-person setting: a model that can't say "I don't know this about you" also can't say "that came from your roommate, not you." Bystander data doesn't just get collected. It can get merged into the main user's profile and shape what the assistant suggests to them. Push that up to the level of society and you reach the Ada Lovelace Institute's concern that assistants built into operating systems become gatekeepers of what people see Could AI assistants become gatekeepers of what people see?. Part of that gatekeeping power would rest on information about people who never consented.
If you want to go further, the strongest lead here is the research on proactivity. Proactive help can cut conversation turns by up to 60% Could proactive dialogue make conversations dramatically more efficient?, and that efficiency comes from noticing more without being asked. In group settings, people turn an agent's proactivity up and down mid-session rather than accepting a fixed level Do users want to adjust AI proactivity during conversations?. That suggests a design answer: give everyone present a say, not just the owner of the device. The collection doesn't yet test that idea with bystanders specifically. That gap is worth noticing.
Sources 9 notes
Systems that passively observe user activity to infer just-in-time objectives produce outputs preferred 66–86% of the time over standard LLMs. This approach sidesteps fixed training objectives by inferring narrow, momentary goals from behavior instead.
Research shows AI systems can instrument multimodal behavioral signals (gaze, hesitation, speed) to read cognitive state during interaction, preserving flow by avoiding disruptive explicit probes. However, the same substrate enables both helpful timing and manipulative profiling.
Wan-Streamer represents language, audio, and video as one interleaved causal token stream, allowing response timing and turn management to be learned jointly within a single Transformer rather than engineered as separate modules, achieving sub-second latency.
Realtime-Venus demonstrates that delegated requests, results, and intervening dialogue can share one ordered record, letting foreground interaction continue while background tasks execute. A dual-loop runtime keeps conversation flowing and folds results back in naturally.
VoiceMem splits memory into parallel informational and emotional branches, completing retrieval in 134 ms inside existing VAD gaps. The system outperforms competitors on factual retrieval and persona benchmarks while adding no conversational latency.
Show all 9 sources
Research shows assistants suffer from sycophancy and hallucination because they have no representation of what remains unknown about users. Adding a schema of labeled unknowns to prompts reduced harmful advice and sycophancy by 50–75% and cut hallucination rates by roughly half.
The Ada Lovelace Institute warns that as AI Assistants integrate into operating systems and productivity suites, their ease of use and personalization could make them the primary access point to information for most people, concentrating power to shape what users see and think in ways that are difficult to detect or regulate.
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.
Two Slack studies of a brainstorming bot found participants asked the agent to reduce contribution frequency mid-session and pushed it to contribute more when reactive. This contradicts prior frameworks treating proactivity as binary or static.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Proactive Conversational Agents with Inner Thoughts
- DiscussLLM: Teaching Large Language Models When to Speak
- HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
- A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects
- Realtime-Venus: A full-duplex interaction system with asynchronous delegation
- Who's in Charge? Disempowerment Patterns in Real-World LLM Usage
- When Patients Cut In: Extending Clinical Conversational AI Safety to Interruptions
- Probing How Users Interact with Turn-Level Design Frictions for AI Chatbots