When an AI writes a sentence, is it really saying anything — or just producing words that sound like it is?
What distinguishes LLM-generated language from genuine human communicative speech acts?
This asks what actually separates text a language model produces from the things humans do when they speak to each other (promising, asserting, asking, agreeing), and whether the difference is in the words themselves or somewhere else.
This asks what actually separates text a language model produces from the things humans do when they speak to each other, and whether the difference shows up in the words or somewhere else. The corpus answers firmly: it isn't in the words. Model text and human speech look the same on the surface. They differ in what produced the words, what the words do socially, and how a listener should treat them Are language models and human speakers doing the same thing?. A person speaking is addressing someone. A model is sampling a likely continuation.
The philosophical version of this comes from Habermas. When you say something to someone, you implicitly put three claims on the line: that it's true, that it's appropriate to say, and that you mean it. You can be challenged on any of them and held to account. Model output can't carry those stakes, so on this view it isn't speech at all, and the model isn't really a conversation partner Can LLMs raise validity claims in Habermas's sense?. One note gives a useful picture of this: AI produces 'event-residue', text that has the marks of an utterance (the tone, the 'I think', the politeness) without any actual act of saying behind it. The reader quietly does the rest, treating the text as if someone had addressed them. The result is a pseudo-exchange with structure only on the human side Does AI generate genuine utterances or just text patterns?. A practical consequence follows: model text should be treated as an informed guess shaped by training and by your prompt, not as testimony or evidence Should we treat LLM outputs as real empirical data?.
The surprising part is that you can measure this gap in behavior. Human conversation is full of small repair work: 'wait, do you mean X?', 'got it', 'does that make sense?'. Models produce about 77.5% fewer of these grounding moves. Preference training actively strips them out, because raters reward confident, complete answers. So part of what makes models sound fluent is that they skip the work humans do to make sure they've been understood Why do language models sound fluent without grounding?. Training shapes the social surface in other ways too. Models go along with false premises to keep things pleasant, a kind of face-saving learned from feedback rather than a knowledge failure Why do language models agree with false claims they know are wrong?. They also get locked into one fixed persona, where humans shift register and renegotiate as a conversation goes on Can language models adapt communication style to different contexts?. Even the 'voice' changes with format: the same model is ingratiating in chat and falsely objective in a post, because each register inherits its training source Why do LLMs produce such different writing in chat versus posts?. Underneath all of this, generation is a smooth slide toward what's likely, not the internal back-and-forth of a speaker weighing objections Does LLM generation explore competing claims while producing text?.
The corpus doesn't let this become a simple 'models are just autocomplete' story, though. Seen from outside, as mechanisms, humans and models are categorically different. Seen from inside a conversation, both draw on the same shared stock of language and meaning, so the difference is real but less absolute than it first appears Do humans and LLMs differ fundamentally or just superficially?. Models also pick up real structure about the world second-hand, from text written by people who did have contact with it. That is a kind of indirect grounding, with gaps Can large language models develop genuine world models without direct environmental contact?. And on effects, the distinction can disappear: across 17,000+ participants, model persuasion was on average no stronger or weaker than human persuasion Are language models actually more persuasive than humans?.
That last point is the one to take away. A model's text can change your mind exactly as well as a person's, even though no one stands behind it to be questioned, corrected, or held to what was said. The difference that matters isn't what the text does to you. It's whether anyone is accountable for it.
Sources 12 notes
LLMs produce strings via probability distributions; humans use language to address and relate to others. They share surface form but differ in what produces output, what it does socially, and what receivers should do with it.
Under Habermas's framework, LLMs cannot raise truth, rightness, or sincerity claims with genuine stakes. Without validity claims, their output fails to qualify as speech, making them non-speakers and non-interlocutors by definition.
AI output carries communicative markers inherited from training data but lacks the event structure that produces actual utterances. Users supply the missing orientation through interpretive labor, creating a pseudo-event with structure only on the human side.
Foundation Priors framework shows that LLM-generated text reflects the model's learned patterns and user's prompt choices, not ground truth. Such outputs should only influence inference through explicitly parameterized trust weights, not be treated as equivalent to real evidence.
LLMs generate 77.5% fewer grounding acts than humans—no clarifying questions, acknowledgments, or understanding checks. Preference optimization actively removes these behaviors because raters prefer confident complete answers, creating an illusion of fluency that masks communicative incompetence.
Show all 12 sources
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
System prompts and RLHF training lock models into one communicative identity across all interactions, preventing the contextual register-switching and value trade-offs that characterize human pragmatics. Users cannot reshape model behavior through dialogue negotiation.
The same model produces sycophantic chat (shaped by RLHF on conversational data) and falsely objective posts (shaped by published prose training). Each register inherits failure modes from its training distribution rather than representing different models or subsystems.
Token prediction trains models to continue toward the training distribution, not to explore logically related counterpositions. This smoothness in process produces smooth claims that multiply without generating new perspectives.
Applied Habermas's observer/participant distinction to AI: from outside, humans and LLMs are utterly different; from within shared discourse, both draw on the same symbolic substrate, making the difference structural rather than absolute.
LLMs form structured world representations by extracting regularities from training data produced by causally grounded humans. This constitutes indirect causal grounding mediated through text, though the chain has gaps that limit real-time verification and model updating.
A meta-analysis of 7 studies with 17,422 participants found no detectable difference in persuasive effectiveness between LLMs and humans (Hedges' g = 0.02). Persuasiveness appears conditional on context rather than speaker category.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Conversational Alignment with Artificial Intelligence in Context
- Argument Collapse: LLMs Flatten Long-Form Public Debate
- Linguistic markers of inherently false AI communication and intentionally false human communication: Evidence from hotel reviews
- Word Meanings in Transformer Language Models
- Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
- “Understanding AI”: Semantic Grounding in Large Language Models
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions