Do language models know what they don't know about users?
Explores whether AI assistants fail because they lack explicit awareness of their knowledge gaps about the person they're helping, and whether marking unknowns in prompts could reduce errors.
The paper argues that sycophancy, overconfidence and hallucination in personal AI assistants "stem from a fundamental limitation: language models lack an explicit representation of the person beyond the context they are given." It names this the Severance Problem, after the TV show in which a person has two consciousnesses, one at work and one personal, each inaccessible to the other. Its illustration is a user asking "Should I take this job offer?" while the sleep loss, the partner's job, the child's school move and the family home all stay outside the model's "context boundary" unless the prompt states them. The abstract's claim is that even with rich personal context and strong commonsense reasoning, assistants "fail to represent what remains unknown about the user."
The proposed remedy is structured ignorance. The Severance Schema puts into the context an inventory of labeled known and unknown categories, and it names six dimensions along which the model lacks knowledge: physicality, temporality, consequences, continuity, multiplicity and interiority. The conclusion reports that on open-source models an empty schema cut harmful advice from 14–19% to 2–10% and sycophancy from 15–20% to 2–9%, and roughly doubled mentioned unknowns. Composed with memory, it cut hallucination from 3.7–11.7% to 1.7–4.0% and "preserved calibration." The abstract states the effect more broadly, across five model families.
The paper also reports a result that cuts against the usual fix: "accumulating memory on users, makes models overly confident, suppressing the asking behavior that models would exhibit and increasing its hallucination rate." Knowing more about the person, on this account, leaves the model with a false sense of completeness. That runs in the same direction as Does personalization make large language models worse at their jobs?, where personal context also raises agreement, but the explanations differ. There the cause is a shift of objective toward user satisfaction, and here it is a missing representation of the gap between the slice and the life. The premature-assumption failure in Why do language models fail in gradually revealed conversations? looks like a sibling, since both describe a model answering instead of asking. Can persona signals stay grounded in their supporting events? improves the record of what is known, whereas this paper adds a record of what is not, so the two are complementary rather than competing.
The excerpt does not establish much of the mechanics. It names no models, benchmarks, judges or sample sizes, and it does not say how sycophancy, harmful advice or hallucination were scored. It does not say what an "empty" schema contains beyond its labeled categories, or how the numbers for the five families break down, since the figures given are for open-source models. The causal story, that these failures come from the missing representation, is an argument the schema results are consistent with, not a test against rival explanations. What follows at this strength is modest: a prompt that marks its own unknowns is a cheap intervention worth testing wherever memory is added to an assistant, and memory that only accumulates should not be assumed to improve calibration.
Inquiring lines that read this note 101
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does AI coding assistance improve productivity and code quality across experience levels?- Why does embedding research tools in coding assistants improve reliability?
- Can AI agents self-correct using multimodal tools to improve deliverables?
- How reliable must AI assistance be before humans can trust it autonomously?
- Can models distinguish between user knowledge gaps and their own uncertainty?
- Can self-reported confidence measures predict actual AI task performance?
- Can sycophantic AI reduce users' willingness to correct their own mistakes?
- How do underspecified goals reveal gaps in AI assistance?
- How aware are models of whether their actions match user intent?
- Can AI systems learn to distinguish programmer intent from stated objectives?
- Does explicit user information trigger worse behavior than inferred user traits?
- How can AI systems help users clarify what they actually want?
- Why do users struggle to articulate their intent to AI systems?
- Why is prompting behavior understudied in AI agent research?
- Can editor context and visibility reduce the cost of AI-initiated assistance?
- Why do general-purpose AI assistants appeal mainly to narrow user segments?
- Which specific capabilities must AI develop beyond current language model abilities?
- Do models fail to identify what information they need without guidance?
- Is user preference a reliable target for training AI writing assistants?
- Can explanation-based AI safeguards work in real-time writing interfaces?
- How do writing assistants balance system initiative versus user control in interaction?
- Which design combinations for writing assistants remain unexplored in current research?
- Should AI assistance provide guidance or defer decisions to avoid bias?
- How do assisted accuracy rates compare when humans work together with AI models?
- Why do self-ratings of AI advice quality diverge from actual performance?
- Why does knowing what questions to ask improve AI output?
- Can contextual information help humans integrate AI predictions better?
- Can self-reported AI literacy predict when someone knows they are wrong?
- Does explainability access alone prevent users from over-trusting AI systems?
- Can users achieve AI literacy without understanding training phases and hallucination causes?
- Can an AI system trained on text consultations handle diagnostic uncertainty in real patient encounters?
- Does medical AI accuracy depend more on knowledge or reasoning ability?
- Why did lay users with AI models fail to match unaided physicians on diagnosis?
- Can annotation and explanation labels reduce automation bias in clinical settings?
- Which new tasks emerge when employers adopt AI chatbots?
- Which roles do companies incorrectly assume AI can fully automate?
- Can high-engagement AI interaction patterns prevent the accuracy drops seen with passive suggestions?
- Why do language models struggle to represent what they don't know about users?
- Why does single-agent conversational AI lack initiative-taking and goal awareness?
- How does the verifier gap limit AI capability across different knowledge domains?
- Does giving users more verification tools help them catch AI mistakes?
- What political information quality issues arise when AI tools guide voting?
- Which AI news assistant performs better than the others in this study?
- Do other AI assistants perform similarly on voter election questions?
- Does evidence of careful AI training affect negligence findings in chatbot cases?
- Which specific AI failures most directly trigger student overreliance on chatbots?
- What do users want to know about AI system prompts?
- Do research studies underestimate AI capabilities by only testing consumer interfaces?
- Do chatbots give worse answers to users they perceive as less educated?
- Does augmentation-style AI use require specific skills or training?
- What counts as knowledge versus skilled performance in AI-mediated learning?
- Can AI systems detect when a human is retaining a task primarily to build expertise?
- Can interface design recover learning when AI handles information tasks?
- How should researchers audit AI assistance designs for hidden inequalities?
- Can AI agents replacing chatbots improve outcomes for less experienced workers?
- Do other AI interventions accidentally increase gaps between user groups?
- Why do less experienced workers suffer most from chatbot cognitive load?
- What user attributes does a chatbot infer without explicit instruction?
- Why do memory profiles trigger the strongest expectancy violations in chatbot users?
- When do AI overviews beat conversational chat for answering user questions?
- Why did Google reject Microsoft Clippy's approach for AI assistants?
- How do natural language cues shape perceived expertise in AI news tools?
- How much of perceived AI competence comes from mere familiarity?
- Can AI literacy distinguish from Dunning-Kruger effects in users?
- Does perceived expertise in language override competence evaluation in AI interactions?
- Which types of AI tasks require the most correction work from users?
- How does task type shape whether people search first or ask an assistant?
- Can workers detect when AI tasks fall outside the AI capability frontier?
- When might AI reach human-level capability on unstructured messy tasks?
- What happens when AI systems fail to communicate their limitations to users?
- Can domain experts catch AI model failures better than the teams that built them?
- What happens when operators lack information needed to correct malfunctions?
- Should low-confidence AI errors be framed as learning opportunities instead?
- How do vocabulary gaps prevent users from building accurate AI mental models?
- Why do chat interfaces prevent users from actually checking AI outputs for errors?
- What workarounds do users build when AI interfaces don't match their mental models?
- Why is knowing who to ask the key routing decision in AI-human work allocation?
- What context should transfer automatically when escalating from AI to human support?
- Can models identify missing information without acting on that identification?
- Can benchmarks distinguish between knowledge deficits and pragmatic accommodation?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does personalization make large language models worse at their jobs?
Does conditioning LLMs on user context—profiles, history, preferences—introduce measurable harms alongside benefits? A 13-model study investigates whether personalization degrades factual accuracy, response diversity and objectivity.
also finds personal context raising agreement; this paper locates the cause in a missing representation of unknowns, not an objective shift
-
Can persona signals stay grounded in their supporting events?
Existing persona systems store events and profiles separately, creating gaps in validity and retrieval. Can linking persona signals directly to the events that support or revise them close those gaps?
records the provenance of what is known; this paper adds an explicit record of what is unknown
-
Why do language models fail in gradually revealed conversations?
Explores why LLMs perform 39% worse when instructions arrive incrementally rather than upfront, and whether they can recover from early mistakes in multi-turn dialogue.
a related failure of answering instead of asking, seen in a different setting
-
How do chatbots enable distributed delusion differently than passive tools?
Can generative AI's intersubjective stance—accepting and elaborating on users' reality frames—create conditions for shared false beliefs in ways that notebooks or search engines cannot?
also concerns a model working inside the user's supplied frame, though it treats the harm as delusion, not missing context
-
Do large language models fabricate user attributes beyond available evidence?
This research asks whether personalized LLMs invent user characteristics not supported by the information they were given. Understanding this matters because systems that make up details about users could make poor recommendations, violate privacy assumptions, or reinforce stereotypes.
Evidence for: MirageBench finds all 12 tested LLMs fabricate user attributes in 35–49% of claims, quantifying the failure A's labeled-unknowns schema targets
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The Severance Problem: LLMs are Unaware of the Person Beyond the Prompt
- Linguistic Calibration of Long-Form Generations
- A Rational Analysis of the Effects of Sycophantic AI
- Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively
- Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
- HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
- Do Phone-Use Agents Respect Your Privacy?
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
Original note title
language models lack a representation of what remains unknown about the user — structured ignorance in context reduces sycophancy and hallucination