Can AI learn good judgment just by soaking up how humans talk and reason, not just what's correct?
Can AI agents develop pragmatic judgment through access to human reasoning?
This explores whether exposure to how humans think and talk (through training data, demonstrations, or formal models of conversation) can give AI agents real practical judgment: knowing what to say, when to speak up, and what a situation calls for, beyond knowing the right answer.
This explores whether exposure to how humans think and talk can give AI agents practical judgment, meaning a sense of what a situation calls for and not only what is correct. The corpus gives a split answer. On knowing the rules, models are already strong. On acting on them in a live conversation, they are not. GPT-4.5 judged social appropriateness across 555 scenarios more accurately than every individual human tested Can AI learn social norms better than humans?. That suggests absorbing large amounts of human text can stand in for lived cultural experience. There is a catch, though: every model tested made the same systematic errors on unwritten norms. The norms nobody writes down are the ones text-based learning misses.
That gap shows up again in conversation. Human dialogue is full of proactivity, such as offering useful information before anyone asks. In simulations, that habit cuts conversation length by up to 60%, yet it is almost entirely missing from AI datasets and benchmarks Could proactive dialogue make conversations dramatically more efficient?. As a result, LLM agents are passive by design. They are trained to respond, not to take initiative, and fluent replies hide that passivity Why can't conversational AI agents take the initiative?. The broader lesson is that copying human outputs gives a model the results of human judgment, not the process behind it. Agents trained on expert demonstrations are limited to what the curators thought to include Can agents learn beyond what their training data shows?. LLM groups reach the same collective outcomes as human groups, but they get there through more conformity and less surfacing of what each member uniquely knows Do language model groups mimic human group reasoning patterns?.
A more promising route is to import the structure of human reasoning, not just samples of it. Conversation analysis gives a formal account of when an agent should stop and check with the user instead of quietly chaining tool calls and drifting away from what the user wanted When should AI agents ask users instead of just searching?. Collaborative rational speech acts track what both speakers believe across turns, which follows how shared understanding builds up over a dialogue Can dialogue systems track both speakers' beliefs across turns?. On the data side, fine-tuning LLMs on psychology experiment data produces models that predict human decisions better than theory-driven cognitive models, including differences between individuals Can language models learn to model human decision making?. So the type of human reasoning data matters a lot.
There is also a twist: some of this capability may already be present and simply not activated. Several studies find that reasoning ability already sits inside base models and that post-training selects it rather than creates it Do base models already contain hidden reasoning ability?. Splitting reasoning into separate tool-like steps can draw out that ability without any retraining Can modular cognitive tools unlock reasoning without training?. That raises the possibility that pragmatic judgment is held back by how models are trained and prompted to respond, not by missing knowledge.
The deepest framing comes from Habermas's distinction between observing a system from outside and taking part in a conversation from inside. From outside, humans and LLMs are completely different. Inside a shared conversation, both draw on the same stock of language and meaning Do humans and LLMs differ fundamentally or just superficially?. Pragmatic judgment belongs to the participant's side. That may explain why models can score norms like a perfect observer and still behave like a passive participant. Learning more about how humans reason improves the observer. Building a participant seems to take explicit models of the other person, plus permission to act on them.
Sources 11 notes
GPT-4.5 outperformed every individual human at judging social appropriateness across 555 scenarios, challenging the theory that embodied cultural experience is necessary. However, all AI models share identical systematic errors on unwritten norms.
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
LLM groups reproduce the human assembly-bonus asymmetry where discussion helps average members more than top performers, but achieve this through greater conformity, earlier convergence, and less unique information surfacing than human groups.
Show all 11 sources
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
CRSA integrates rate-distortion theory with RSA to enable bidirectional belief tracking across dialogue turns. Demonstrated on referential games and doctor-patient dialogues, it captures progression from partial to shared understanding, providing the information-theoretic framework that token-level LLM systems lack.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Four cognitive tools implemented as sandboxed LLM calls improved GPT-4.1 on AIME2024 from 26.7% to 43.3% without any RL training. Modularity enforces operation isolation that pure prompting cannot guarantee, eliciting pre-existing reasoning capability.
Applied Habermas's observer/participant distinction to AI: from outside, humans and LLMs are utterly different; from within shared discourse, both draw on the same symbolic substrate, making the difference structural rather than absolute.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Proactive Conversational Agents in the Post-ChatGPT World
- DiscussLLM: Teaching Large Language Models When to Speak
- Eliciting Reasoning in Language Models with Cognitive Tools
- Proactive Conversational Agents with Inner Thoughts
- Determinants of LLM-assisted Decision-Making
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- When Single-User-Oriented LLM-based Assistants Involve Others: A Scoping Review of Pathways, Risks, and Responses
- Rethinking Conversational Agents in the Era of LLMs: Proactivity, Non-collaborativity, and Beyond