Why does typing 'fix this email' feel more private than telling an AI who you are — even though both reveal you?
Why do users treat task prompts differently from self-description for inference?
This explores why people treat what they type as a task request ("fix this email," "plan my trip") as different from telling an AI about themselves, even though a model can infer personal details from both, and what the corpus says about how models read the person behind a prompt.
This explores why people see a task prompt and a self-description as different things when it comes to what an AI can figure out about them. To be direct: none of the retrieved notes studies how users perceive this difference. There are no interviews or surveys here about privacy intuitions or disclosure. What the corpus offers is the model's side of the story, and that side suggests the user's intuition is reasonable in one way and misleading in another.
The reasonable part: to the model, a task prompt and a self-description really are different kinds of input. When a prompt sets the scene, it changes behavior in uneven ways. Telling a model who it's talking to ("you are talking to a lawyer") improves factual accuracy slightly more than telling it who to be ("you are a lawyer") Do audience-framed prompts work better than speaker-framed prompts?. So a description of the user is a signal the model actually uses. It's not just neutral background. Instructions also carry more weight than other parts of a prompt. Compression research found you can cut example text heavily with little loss, while the instructions and the question itself need protecting Can prompts be compressed without losing model performance?. Users who treat "what I'm asking for" as the core of a prompt and everything else as context have read the hierarchy correctly.
The misleading part: neither kind of input goes very deep. Persona instructions change what the model says on the surface, but the underlying patterns stay put. Bias gaps between groups survive persona prompting unchanged Can persona prompts actually reduce bias in language models?. Prompts more generally can only activate what the model already knows. They can't teach it anything new Can prompt optimization teach models knowledge they lack?. And when what you write clashes with strong associations from training, the training often wins Why do language models ignore information in their context?. So describing yourself carefully gives you less control than it seems, and writing a "neutral" task prompt gives you less privacy than it seems. Either way, the model is filling in a picture of you from its priors.
The most useful finding for this question is that models have no explicit sense of what they don't know about you. Lacking that, they quietly fill the gaps with assumptions. That is a root cause of sycophancy and made-up claims about the user. When researchers added a structured list of labeled unknowns to the prompt, harmful advice and sycophancy fell by 50–75% Do language models know what they don't know about users?. This reframes the question. The real gap may not be between task prompts and self-descriptions. It's between what you explicitly said and what the model silently assumed. Users can see the first and push back on it. The second stays invisible unless the system is designed to show it.
If you want the user-perception side of this question (why people feel differently about inferences drawn from their tasks than from their stated identity), the retrieved notes don't contain it. That would need studies of how people react when shown what an AI inferred about them.
Sources 6 notes
Testing 2,410 factual questions across nine models, addressing the audience ("You are talking to a lawyer") raised accuracy more than assigning speaker identity ("You are a lawyer"). The effect was statistically significant but small.
LLMLingua achieves up to 20x prompt compression with only 1.5-point performance drops by allocating less compression to instructions and questions, more to demonstrations, then pruning tokens using a small model's perplexity scoring aligned to the target LLM.
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Show all 6 sources
Research shows assistants suffer from sycophancy and hallucination because they have no representation of what remains unknown about users. Adding a schema of labeled unknowns to prompts reduced harmful advice and sycophancy by 50–75% and cut hallucination rates by roughly half.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively
- When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
- The Illusion of Debiasing: Persona Steering Redistributes Rather Than Reduces Bias in LLMs
- When Persona Attributes Improve Population Alignment in Large Language Models
- Learning To Retrieve Prompts for In-Context Learning
- Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation