SYNTHESIS NOTE
Topics›Assistants Personalization›this note

Do language models know what they don't know about users?

Explores whether AI assistants fail because they lack explicit awareness of their knowledge gaps about the person they're helping, and whether marking unknowns in prompts could reduce errors.

Synthesis note · 2026-09-25 · sourced from Assistants Personalization

The paper argues that sycophancy, overconfidence and hallucination in personal AI assistants "stem from a fundamental limitation: language models lack an explicit representation of the person beyond the context they are given." It names this the Severance Problem, after the TV show in which a person has two consciousnesses, one at work and one personal, each inaccessible to the other. Its illustration is a user asking "Should I take this job offer?" while the sleep loss, the partner's job, the child's school move and the family home all stay outside the model's "context boundary" unless the prompt states them. The abstract's claim is that even with rich personal context and strong commonsense reasoning, assistants "fail to represent what remains unknown about the user."

The proposed remedy is structured ignorance. The Severance Schema puts into the context an inventory of labeled known and unknown categories, and it names six dimensions along which the model lacks knowledge: physicality, temporality, consequences, continuity, multiplicity and interiority. The conclusion reports that on open-source models an empty schema cut harmful advice from 14–19% to 2–10% and sycophancy from 15–20% to 2–9%, and roughly doubled mentioned unknowns. Composed with memory, it cut hallucination from 3.7–11.7% to 1.7–4.0% and "preserved calibration." The abstract states the effect more broadly, across five model families.

The paper also reports a result that cuts against the usual fix: "accumulating memory on users, makes models overly confident, suppressing the asking behavior that models would exhibit and increasing its hallucination rate." Knowing more about the person, on this account, leaves the model with a false sense of completeness. That runs in the same direction as Does personalization make large language models worse at their jobs?, where personal context also raises agreement, but the explanations differ. There the cause is a shift of objective toward user satisfaction, and here it is a missing representation of the gap between the slice and the life. The premature-assumption failure in Why do language models fail in gradually revealed conversations? looks like a sibling, since both describe a model answering instead of asking. Can persona signals stay grounded in their supporting events? improves the record of what is known, whereas this paper adds a record of what is not, so the two are complementary rather than competing.

The excerpt does not establish much of the mechanics. It names no models, benchmarks, judges or sample sizes, and it does not say how sycophancy, harmful advice or hallucination were scored. It does not say what an "empty" schema contains beyond its labeled categories, or how the numbers for the five families break down, since the figures given are for open-source models. The causal story, that these failures come from the missing representation, is an argument the schema results are consistent with, not a test against rival explanations. What follows at this strength is modest: a prompt that marks its own unknowns is a cheap intervention worth testing wherever memory is added to an assistant, and memory that only accumulates should not be assumed to improve calibration.

Inquiring lines that read this note 101

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Does AI coding assistance improve productivity and code quality across experience levels? How do confidence signals and expertise language mislead users' trust calibration? How reliably can LLM-based persona simulations predict human behavior? How does personalization affect trust and privacy concerns together? How can AI systems be proactive while respecting user intent? Do language models rely on encoded knowledge or primarily on surface patterns? When readers detect AI involvement, does it undermine writer credibility? Can employers detect AI-assisted applications and should they be permitted? How can evaluation systems stay reliable and resistant to AI-driven gaming? When does AI explainability improve versus hinder human oversight and trust? How do clinicians assess, trust, and effectively use medical AI in practice? How does AI exposure shape employment demand and worker earnings? How can conversational AI develop appropriate initiative and goal awareness? What organizational factors determine whether firms successfully adopt and integrate AI? How do standardized benchmarks differ from measures of real-world AI capability? Can sequences of locally safe agent actions violate system-level constraints? How can verification systems keep pace with AI generation? Can AI systems achieve sustainable self-improvement without external human guidance? How do chatbot properties shape user belief adoption and resistance? How does AI assistance affect professional skill development and retention? How does AI assistance affect human cognitive skill development over time? How do users build accurate mental models when agent reasoning is partially hidden? How does AI adoption reshape workplace inequality and labor market outcomes? Does personalization in chatbots strengthen engagement or undermine authentic interaction? How do AI-generated summaries affect traffic and engagement with original sources? Does polished AI presentation mislead people about accuracy? Which task types benefit from human-AI collaboration and how should work divide? Why do users misjudge the credibility of AI-generated and verified content? How does interaction history visibility influence agent coordination and strategic behavior? Can warnings effectively reduce AI's persuasive influence on audiences? What happens to the meaning and authority of expertise when AI generates knowledge? Should AI systems make failures transparent so users can contest and adjust? Can prompting inject new knowledge into models or only activate existing capabilities? How should control and autonomy be allocated in human-AI workplace collaboration? How are knowledge and reasoning functionally separated in neural network architecture? How does interface design shape effectiveness in human-AI collaboration? Is mimicking human communication the best design approach for conversational AI? How aligned are models' verbal explanations with their actual internal reasoning? What shapes user privacy concerns about AI agent inference and memory? Do language models learn genuine understanding or just surface patterns? Can prompt engineering achieve genuine reasoning improvements without introducing user bias? Can external memory modules outperform model weight consolidation for long-term retention?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 142 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

language models lack a representation of what remains unknown about the user — structured ignorance in context reduces sycophancy and hallucination