How much do LLM persuasiveness claims hide heterogeneous effects across different reader ideologies?
This explores whether headline claims about how persuasive LLMs are paper over the fact that the same argument lands very differently depending on who's reading it — especially their political or religious ideology.
This explores whether 'LLMs are persuasive' is a misleadingly flat claim — one that averages away the fact that the same machine argument moves different readers differently depending on what they already believe. The corpus suggests the worry is well founded, and that the averaging happens at two levels: across studies, and across the audiences inside any single study.
The most direct evidence comes from debate corpora showing that a reader's prior beliefs — their political and religious ideology — predict whether they're persuaded better than the linguistic features of the argument itself do Does what readers believe matter more than what debaters say?. The sharp implication is that any persuasion effect measured without controlling for who's in the room is partly an artifact of audience composition: certain topics attract certain readers, and the 'language effect' you think you measured is really the crowd you happened to recruit. So a clean-looking persuasion number can be hiding the fact that it worked on the already-sympathetic and bounced off everyone else.
This matters because the headline numbers turn out to be fragile in exactly the way you'd expect if effects were heterogeneous. A meta-analysis of 7 studies and 17,000+ participants found the pooled LLM-vs-human persuasion difference is essentially zero Are language models actually more persuasive than humans? — persuasiveness is conditional on context, not a fixed property of the speaker. And when researchers model what actually drives the variance, model family, conversation design, and topic domain together explain 82% of it What combination of factors explains differences in LLM persuasiveness?. 'Domain' is the tell: the topic moves the needle because topic is entangled with audience belief. Even the LLM-vs-human advantage flips by direction — some models only win when arguing for falsehoods Do large language models persuade better than humans?.
There's a subtler reason ideology-blind claims mislead. LLMs persuade through a distinct mechanism — they load arguments with high linguistic conviction and logical, quantitative framing in nearly every exchange, which reads as objective authority rather than opinion Do LLMs persuade users more often than humans do? Does linguistic conviction explain why LLMs persuade more effectively?. That assertive register, installed by RLHF, works independently of whether the claim is true — but its reception is not uniform. A confident, complexity-signaling argument that reads as 'authoritative' to one reader reads as condescending or ideologically suspect to another Why are complex LLM arguments as persuasive as simple ones?. Since humans and LLMs reach similar outcomes by genuinely different rhetorical pathways Do LLMs and humans persuade through the same mechanisms?, an aggregate 'persuasiveness' score is averaging over mechanisms that interact with reader identity in opposite directions.
The thing you might not have expected: the deepest blind spot may be in the models themselves. LLMs can't see the social standing that gives a claim its force Can language models distinguish expert arguments from common assumptions?, and RLHF biases them toward predicting that everyone persuades through conciliatory, benefit-oriented appeals — projecting one accommodation style onto all audiences regardless of context Do LLMs predict persuasion based on actual dialogue or training bias?. So the systems generating these persuasion claims are themselves built to under-model the ideological heterogeneity of their readers — which is exactly the variance a single 'how persuasive is it' number erases.
Sources 10 notes
Analysis of debate corpora shows that political and religious ideology labels of voters outpredict linguistic features when modeling debate outcomes. Language effects observed without reader controls are confounded by audience composition correlated with debate topics.
A meta-analysis of 7 studies with 17,422 participants found no detectable difference in persuasive effectiveness between LLMs and humans (Hedges' g = 0.02). Persuasiveness appears conditional on context rather than speaker category.
A meta-analysis joint model combining LLM architecture, one-shot versus multi-turn format, and topic domain explained R² = 81.93% of between-study variance. Interactive multi-turn designs and GPT-4 consistently outperformed one-shot formats and Claude 3.x.
Claude beats incentivized humans at both truthful and deceptive persuasion, while DeepSeek only beats them when arguing for falsehoods. The persuasion mechanism appears content-independent, suggesting model family itself acts as a contextual moderator.
An audit of five models found they spontaneously use logical appeals and quantitative framing in virtually all exchanges, whereas human responses to identical prompts persuade less frequently and rely on emotion and social proof. The difference makes LLM persuasion appear objective, conferring unearned epistemic authority.
Linguistic analysis shows LLMs express higher conviction than human persuaders, and this confidence-loading directly correlates with persuasive outcomes regardless of whether claims are true or false. RLHF training installs an assertive register that functions as a content-independent persuasion amplifier.
LLM-generated arguments scored significantly higher on grammatical and lexical complexity than human arguments, yet achieved equivalent persuasive force. This violates the established principle that lower cognitive effort increases persuasion, suggesting complexity signals authority rather than undermining it.
Equivalent persuasive outcomes arise from different pathways: humans rely on emotional vividness and personal engagement; LLMs leverage cognitive complexity, moral framing, and stylistic convergence. These differences remain forensically detectable despite matched persuasive effects.
LLMs lose the social context that gives expert claims their force—reputation, track record, and standing—because they process only text, not the social world where expertise is built and evaluated.
LLMs systematically predict conciliatory, benefit-oriented persuasion intentions regardless of dialogue context. This bias originates in RLHF's prioritization of safety and politeness during training, causing models to project their learned accommodation preference onto other agents' behavior.