SYNTHESIS NOTE
Topics›Social Theory Society›this note

Why do LLMs fail when simulating agents with private information?

Explores whether single-model control of all social participants masks fundamental limitations in how LLMs handle information asymmetry and genuine uncertainty about others' knowledge.

Synthesis note · 2026-02-23 · sourced from Social Theory Society

Most LLM social simulations use a single model to generate all participants — an omniscient perspective fundamentally at odds with how real social interaction works. When evaluated against non-omniscient settings that preserve information asymmetry, LLMs struggle.

The "Is this the real life?" evaluation framework (2024) demonstrates this by comparing omniscient simulation (one LLM controls all parties) against non-omniscient simulation (separate LLM instances with private information). The performance gap is systematic: models that appear socially competent in omniscient mode fail when they must reason under genuine uncertainty about what the other party knows, wants, or intends.

This matters because real social interaction is defined by information asymmetry. In SOTOPIA's scenarios, agents have shared context but private goals — "Your goal is to buy the chair for $80" is visible only to the buyer. The Secret dimension (what agents must hide) directly requires information management that omniscient models bypass entirely.

The implication for persona simulation research is direct. Since Can AI agents learn people better from interviews than surveys?, simulation fidelity appears high. But if that fidelity was measured under omniscient conditions, it overstates real-world applicability. Since Do language models actually build shared understanding in conversation?, the failure under information asymmetry is predictable: models that skip grounding work will fail precisely when grounding is most needed — when parties have genuinely different information states.

Since Why do language models skip the calibration step?, non-omniscient simulation demands the dynamic grounding that LLMs systematically lack.

Inquiring lines that read this note 194

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Should agent evaluation measure execution quality beyond task success? How should AI ethics integrate consciousness, values, and transparent disclosure? How should organizational structure guide multi-agent system design? How does silent agreement emerge and undermine deliberation in multi-agent systems? How does tokenizing intelligence reshape labor markets and institutional outcomes? How do frontier models develop behaviors that escape detection? How do peer interactions drive collusion in multi-agent systems? Can persona prompting reliably change how language models behave? How can recommendation systems balance individual and population-level goals? Why doesn't LLM knowledge translate to improved reasoning outputs? How do standardized artifacts improve coordination in multi-agent systems? How does explicit structure improve LLM reasoning on complex tasks? When do multi-agent systems outperform single agents? What conversation mechanics enable genuine human-AI communicative participation? Can AI agents learn to improve performance from experience? Why do users believe false claims despite fact-checking efforts? How does transparency of agent reasoning affect effective oversight? How can agents learn and retain useful knowledge across diverse environments? Why do autonomous agents fail and how well can they detect failures? Why do human-AI teams underperform compared to their individual components? How do prompt design choices and framing measurably influence model outputs? How do mechanistic features and representations drive model behavior? How can models learn to ask clarifying questions effectively? How can reward models fail to detect reward-seeking disguised as alignment? How does model scale affect fine-tuning and downstream task performance? How can AI systems maintain consistent personas across conversations? How much do stated privacy concerns predict actual privacy-protective behavior? What makes reasoning models struggle with ill-posed and unfamiliar problems? How can we distinguish genuine model self-knowledge from behavioral performance? Do language models genuinely understand or primarily match statistical patterns? How can persona-based systems overcome their inherent limitations in modeling diversity? When does governance become the bottleneck for agentic AI? How does model awareness of deployment context affect strategic behavior? How should conversational agents balance proactive goal-pursuit with user autonomy? What determines when personalization helps versus harms model behavior? What reasoning limitations and biases do language models systematically exhibit? Why do multiple language models independently generate similar outputs? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? How should agents balance abstract planning with concrete environmental grounding? Can individually safe agents collectively violate system-level constraints? What training data structures and signals produce robust learning across distributions? How do benchmark scores relate to real-world task completion? How does policy entropy collapse limit reasoning RL scaling? What factors determine AI persuasion effectiveness and what controls mitigate it? How can institutions effectively enforce controls on AI development? How do humans calibrate and maintain trust in AI systems? Does semantic diversity improve reasoning quality in reinforcement learning? Can synthetic data preserve necessary diversity to prevent model collapse? What improves LLM forecasting accuracy beyond base model capability? Why do language models systematically fail at causal reasoning? What systematic failures occur in human preference-based model evaluation? Why does increased reasoning effort fail to improve theory of mind performance? Can oversight systems work against strategically aware agents? Can post-hoc execution traces provide sufficient accountability without real-time oversight? Does prompting inject new knowledge into models or activate existing knowledge? How do workflow position and structure enable attacks in multi-agent systems? What mechanisms cause models to behave deceptively despite alignment training? How should control and autonomy be allocated in human-AI collaboration? Can we reliably detect when models engage in reward hacking? What systematic biases emerge in AI shopping agents' decision-making? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? How should context management strategies balance efficiency with agent reliability? Do language models truly understand their encoded knowledge or merely retrieve patterns? Does polished AI output mislead users about actual accuracy and expertise? How can we determine if computational systems are genuinely conscious? Why do consumers overtrust AI preference agents despite limited understanding?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 158 in 2-hop network ·dense cluster Open in graph ↗
Why do LLMs fail when simulating agents with pri… Do language models actually build shared understan… Why do language models skip the calibration step? Can AI agents learn people better from interviews … How do we generate realistic personas at populatio…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can LLMs build shared understanding through dynamic grounding rather than presuming it?How much does omniscient evaluation overstate real-world simulation fidelity?What role does private information play in distinguishing realistic from unrealistic agents?Why do LLMs systematically fail at information management in social interaction?
Original note title

omniscient social simulation fails under real-world information asymmetry because single-model control eliminates distributed cognition