Do language models judge persuasion the way humans do?

The paper tests whether LLMs update beliefs the way humans do, using a naturally occurring online persuasion corpus in which original posters "explicitly verify whether a reply changed their view." Against those human-verified outcomes, every tested model reaches only slight agreement (Cohen's κ from 0.079 to 0.178). The overall divergence rate is similar across models, but "its internal composition differs markedly," so the models fail in different places even when they fail about equally often.

The paper diagnoses the gap through the argument rather than the topic. Humans and LLMs agree on the strongest persuasion cues and split on finer ones: humans are more swayed by novel content and assertive language, while LLMs favor topical similarity and surface-level formatting. At the level of strategy, LLMs underweight emotional appeals and overweight credibility signals. The type of proposition under debate has no measurable effect on divergence, so the mismatch tracks how an argument is built, not what it is about. The authors read this as "a structural mismatch between LLM and human belief updating."

Most neighboring notes study LLMs as persuaders, and this one studies them as persuadees. The producer-side asymmetry is that LLMs persuade through analytical routes and humans through affective ones, as in Do humans and AI persuade through different cognitive routes?. The direction repeats on the receiving side, since the models discount emotional engagement. It does not follow that the two findings share a mechanism: "credibility signals" and "topical overlap" are not the same thing as analytic reasoning, and this excerpt does not test the ELM. The same split between outcome and process appears in Do LLMs and humans persuade through the same mechanisms?, where similar effects come from different ingredients. Here the agreement is weak even on the outcome. The multi-turn result in Can models abandon correct beliefs under conversational pressure? measures sustained pressure on held beliefs, while this paper measures verdicts on a single reply, so neither predicts the other on this evidence. The paper also finds that switching from first-person role-playing to third-person observation "shifts all models toward greater resistance to persuasion." That effect varies by strategy and textual feature, so the framing of the judgment is itself a lever on the output.

The excerpt is silent on several things. It does not give the number or identity of the models, the corpus size, or effect sizes for the cue contrasts. It does not say whether the models over-predict or under-predict persuasion relative to humans in the first-person condition. And it does not say whether the divergence comes from how models represent arguments or from how they are prompted to judge them. The authors leave open whether scaling resolves the gap, noting only that the mismatch "persists even in the strongest model tested." What the evidence supports is a narrow caution. Where an LLM stands in for a human who is being persuaded, as in the social simulations the paper cites, its verdicts need checking against human-verified outcomes and cannot be assumed to track them.