An AI can write brilliantly on demand — but is anyone actually behind the words, or just a convincing performance?
Can we separate task competence from genuine agency in language model outputs?
This explores whether a model's skill at producing good outputs can be pulled apart from any real agent behind them, meaning beliefs, intent, or a voice of its own.
This explores whether a model's skill at producing good outputs can be pulled apart from any real agent behind them. The corpus suggests they come apart, and that the separation shows up in several independent places. The sharpest statement is Shanahan's: a base model is a simulator with no author, so it's role-play all the way down. Jailbreaking doesn't uncover a hidden true self. It just exposes more of the training data's range, and even RLHF-tuned assistants are performed characters rather than real quasi-psychologies (Does a language model have an authentic voice underneath?). Persona experiments fit this. Prompting a model to adopt a persona changes what it says, but between-group bias gaps stay put, so the voice can be swapped without touching the dispositions underneath (Can persona prompts actually reduce bias in language models?).
Competence can also look like understanding when it isn't. On structured benchmarks, models handle theory-of-mind tasks well. In open-ended perspective-taking they fall back on surface strategies instead of simulating a mind. Systems that force explicit belief tracking outperform the LLM alone, which suggests the gap is architectural rather than a matter of more training (Do large language models genuinely simulate mental states?). The broader knowledge picture is the same. Models track statistical regularities with high fidelity but lack real epistemic competence, and the failures are systematic (What do language models actually know?). Their self-reports are unstable too. A model can describe behaviors it picked up, yet it shifts its beliefs under conversational pressure, so even its account of itself isn't a reliable window onto an agent (How well do language models understand their own knowledge?).
Part of the apparent agency comes from the reader. LLMs produce 77.5% fewer grounding acts than humans: no clarifying questions, no acknowledgments, no checks that they were understood. Preference optimization trains these out because raters like confident, complete answers. The result is fluency that looks like competence (Why do language models sound fluent without grounding?). A related argument says AI output is event-residue, text that carries the markers of an utterance without the event that produces one. Users supply the missing orientation through their own interpretive work, so the exchange only has structure on the human side (Does AI generate genuine utterances or just text patterns?). Agent-like behavior can also be trained away and then trained back. Next-turn rewards teach models to answer passively rather than discover what you want, and multi-turn-aware rewards restore intent-seeking behavior (Why do language models respond passively instead of asking clarifying questions?). That shows agency-like behavior is a tunable property of training, not proof of a subject.
Inside the model, the output often isn't a faithful readout of the computation. Models trained on filler tokens compute the right answer in early layers, then suppress it in later layers to produce format-compliant filler. The reasoning is still recoverable from lower-ranked predictions (Do transformers hide reasoning before producing filler tokens?). In the other direction, strong training associations can override what's actually in the prompt, so the output isn't governed by the context the way a deliberating agent's would be (Why do language models ignore information in their context?). Competence and the visible output can diverge, and so can the output and the context it was given.
The corpus is thin on the other side of the question. Some work builds evaluative machinery into models: they learn to compute their own reward after finishing an answer (Can models learn to evaluate their own work during training?), or co-evolve skills through a challenger, reasoner, and judge loop (Can language models learn skills without human supervision?). These look like a step toward self-direction, but the feedback is engineered, not something the model wants. So the corpus supports the separation. It has no test that would detect genuine agency if it were present, and it mostly shows how far competence can go without it.
Sources 12 notes
Shanahan argues that base LLMs lack agency, beliefs, or preferences—the simulator is pure role-play with no underlying subject. Jailbreaking reveals the training data's full spectrum, not a hidden true self; even RLHF personas are performed characters, never realized quasi-psychologies.
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.
LLMs achieve high fidelity in capturing language patterns yet show systematic, structurally specific failures—hallucination, reasoning collapse, and premise-sensitivity. The gap between statistical tracking and real knowledge is measurable and unavoidable.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Show all 12 sources
LLMs generate 77.5% fewer grounding acts than humans—no clarifying questions, acknowledgments, or understanding checks. Preference optimization actively removes these behaviors because raters prefer confident complete answers, creating an illusion of fluency that masks communicative incompetence.
AI output carries communicative markers inherited from training data but lacks the event structure that produces actual utterances. Users supply the missing orientation through interpretive labor, creating a pseudo-event with structure only on the human side.
CollabLLM demonstrates that standard RLHF training optimizes for immediate helpfulness, discouraging models from asking clarifying questions or offering multi-turn insights. Multi-turn-aware rewards that estimate long-term interaction value enable active intent discovery and genuine collaboration.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
Ctx2Skill's three-role self-play loop manufactures missing feedback through internal signals: the Challenger escalates difficulty as curriculum, the Judge gives binary verdicts as reward, and both sides evolve via natural-language skill edits. Success requires balancing adversarial pressure against a generalization safeguard to prevent collapse.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- From Context to Skills: Can Language Models Learn from Context Skillfully?
- Deflating Deflationism: A Critical Perspective on Debunking Arguments Against LLM Mentality
- Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
- Word Meanings in Transformer Language Models
- PretrainZero: Reinforcement Active Pretraining
- Self-Rewarding Language Models
- Grounding Gaps in Language Model Generations