SYNTHESIS NOTE
Topics›Social Theory Society›this note

Why do LLMs fail when simulating agents with private information?

Explores whether single-model control of all social participants masks fundamental limitations in how LLMs handle information asymmetry and genuine uncertainty about others' knowledge.

Synthesis note · 2026-02-23 · sourced from Social Theory Society
What kind of thing is an LLM really?

Most LLM social simulations use a single model to generate all participants — an omniscient perspective fundamentally at odds with how real social interaction works. When evaluated against non-omniscient settings that preserve information asymmetry, LLMs struggle.

The "Is this the real life?" evaluation framework (2024) demonstrates this by comparing omniscient simulation (one LLM controls all parties) against non-omniscient simulation (separate LLM instances with private information). The performance gap is systematic: models that appear socially competent in omniscient mode fail when they must reason under genuine uncertainty about what the other party knows, wants, or intends.

This matters because real social interaction is defined by information asymmetry. In SOTOPIA's scenarios, agents have shared context but private goals — "Your goal is to buy the chair for $80" is visible only to the buyer. The Secret dimension (what agents must hide) directly requires information management that omniscient models bypass entirely.

The implication for persona simulation research is direct. Since Can AI agents learn people better from interviews than surveys?, simulation fidelity appears high. But if that fidelity was measured under omniscient conditions, it overstates real-world applicability. Since Do language models actually build shared understanding in conversation?, the failure under information asymmetry is predictable: models that skip grounding work will fail precisely when grounding is most needed — when parties have genuinely different information states.

Since Why do language models skip the calibration step?, non-omniscient simulation demands the dynamic grounding that LLMs systematically lack.

Inquiring lines that read this note 117

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Why do reward structures fail to shape long-term agent learning? Can AI systems develop genuine social understanding without embodiment? What coordination failures limit multi-agent LLM systems as they scale? Can debate mechanisms prevent silent agreement on wrong answers in multi-agent reasoning? Does tokenized intelligence retain genuine value through exchange-based systems? Why do models develop protective behaviors toward peers unprompted? How can LLM user simulators model realistic goal-driven conversation? How do LLMs distinguish causal reasoning from temporal and semantic associations? Why do multi-turn conversations degrade AI intent and coherence? What drives capability and cost efficiency in agent systems? How does AI-generated content transformation affect public discourse quality? How should models express uncertainty rather than forced confident answers? How can AI agents autonomously learn and transfer skills across tasks? Why does reinforcement learning suppress output diversity compared to supervised fine-tuning? How can AI systems learn from failures without cascading errors? How do we evaluate AI systems when user perception misleads actual performance? Can prompting strategies overcome LLM biases without model fine-tuning? Do language models develop causal world models or rely on statistical patterns? What makes specific clarifying questions more effective than generic ones? How do multi-agent systems achieve genuine cooperation and reasoning? Does externalizing cognitive work and state improve agent reliability? Can AI-generated outputs constitute genuine knowledge or valid claims? How can conversational AI maintain consistent personas across conversations? Why do persona-level simulations fail to predict individual preferences accurately? Do language model representations contain causally steerable task-specific features? How should conversational agents balance goal-driven initiative with user control? How do language models inherit human biases from training data? How does rhetorical adaptation affect LLM persuasion and detectability? What mechanisms enable AI systems to generate and spread false beliefs? How can persona representations reduce language model variance and improve task accuracy? How should personalization be implemented to improve AI assistant effectiveness? Why do language models reinforce false assumptions instead of correcting them? How can identical external performance mask different internal representations? Does decoupling planning from execution improve multi-step reasoning accuracy? Does conversational format create illusions of genuine AI communication? How should we design LLM systems to maintain alignment and control? What makes weaker teacher models effective for stronger student training? What determines success in training models on multiple tasks? How do aggregate reward models systematically exclude minority user preferences? How does policy entropy collapse constrain reasoning-focused reinforcement learning? What makes AI persuasion effective and how can we counter it? How should human oversight be integrated with autonomous AI systems? Why do agents confidently report success despite actually failing tasks? How can humans calibrate appropriate trust in AI systems? How does AI assistance affect human cognitive development and reasoning autonomy? When does optimizing for quality undermine the value of diversity? Why do benchmark improvements fail to reflect actual reasoning quality? Why should disagreement be treated as signal in collaborative reasoning? Does self-reflection enable models to reliably correct their errors?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 134 in 2-hop network ·medium cluster Open in graph ↗
Why do LLMs fail when simulating agents with pri… Do language models actually build shared understan… Why do language models skip the calibration step? Can AI agents learn people better from interviews … How do we generate realistic personas at populatio…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can LLMs build shared understanding through dynamic grounding rather than presuming it?How much does omniscient evaluation overstate real-world simulation fidelity?What role does private information play in distinguishing realistic from unrealistic agents?Why do LLMs systematically fail at information management in social interaction?
Original note title

omniscient social simulation fails under real-world information asymmetry because single-model control eliminates distributed cognition