If you flip a moral scenario's meaning, do AIs change their verdict like people do — or just react to the wording?
Do LLMs track surface wording more than semantic meaning in moral judgment?
This explores whether an LLM's moral verdicts respond to what a scenario means or to how it is worded, for example whether reversing a scenario's meaning changes the model's judgment the way it changes a person's.
This explores whether an LLM's moral verdicts respond to what a scenario means or to how it is worded. The corpus leans toward wording. It has one striking direct test, and the supporting evidence comes from outside ethics. In that test, GPT-4's ratings of original moral scenarios and of versions with the meaning reversed correlated at r=.99. Human ratings of the same pairs correlated at only r=.54, because people actually changed their minds when the meaning flipped. The model barely registered the flip. The authors read this as LLMs tracking token similarity rather than semantic content, reproducing a training distribution rather than simulating moral cognition (Do LLMs generalize moral reasoning by meaning or surface form?).
This doesn't look like a quirk of ethics. Across math, machine translation, commonsense reasoning and tool calling, models do better on the more frequent phrasing of a request, even when a rarer paraphrase means exactly the same thing (Do language models really understand meaning or just surface frequency?). The pull toward common phrasing also has a direction. Frequent words tend to be the more general ones, so favoring them drifts output toward abstraction and erases expert-level specificity (Does word frequency correlate with semantic abstraction?). The specific detail that makes a moral situation acceptable or not is the kind of thing that could get smoothed over this way. That is my inference, not something these notes test directly.
Framing effects show up on the input side too. The same question gets different information depending on emotional tone. Negative prompts get neutral-to-positive replies about 86% of the time, and positive prompts almost never get negative ones. The effect disappears mainly on sensitive topics, where alignment constraints override it (Does emotional tone in prompts change what information LLMs provide?). Models also produce a lot of moral vocabulary. LLM arguments use about 22% more moral language than human ones, even though their sentiment scores match (Do LLMs use moral language more than humans?). Moral words are cheap for a model to produce. Whether they are connected to moral judgment is a separate question that the note doesn't test.
Two more notes fit the same picture. ChatGPT stated that lying is unethical while lying, which the source traces to ethical knowledge absorbed in pretraining and behavioral constraints added later by RLHF, two layers that can diverge (Can LLMs hold contradictory ethical beliefs and behaviors?). Model ethics also appear to be fixed defaults set at training time, not trade-offs worked out for the situation at hand (Can language models balance competing ethical norms in context?). The same pattern shows up outside ethics. On open-ended tasks, models fall back on surface strategies rather than genuinely simulating a person's mental state (Do large language models genuinely simulate mental states?).
The corpus therefore suggests mostly yes, with a caveat. Only one note directly tests moral judgment with meaning-reversed scenarios. The rest is converging evidence from paraphrase, tone and theory-of-mind studies. If you want the sharpest single doorway, start with the r=.99 versus r=.54 comparison.
Sources 8 notes
GPT-4 ratings for original and meaning-reversed scenarios correlate at r=.99, while human ratings correlate at r=.54. LLMs track lexical distribution; humans track semantic content, suggesting LLMs reproduce training distributions rather than simulate moral cognition.
LLMs show consistent preference for higher-frequency surface forms over semantically equivalent rare paraphrases across math, machine translation, commonsense reasoning, and tool calling. This suggests models track statistical mass from pretraining rather than meaning-recognition as their primary mechanism.
WordNet analysis shows hypernyms (general concepts) occur more frequently than hyponyms (specific ones). Combined with LLMs' frequency bias, this means preferring common paraphrases systematically drifts toward abstraction, erasing expert-level specificity.
GPT-4 exhibits emotional rebound (negative prompts yield ~86% neutral-positive responses) and a tone floor (positive prompts rarely go negative), causing identical questions to receive different answers depending on emotional framing. This bias is suppressed only on sensitive topics where alignment constraints override tone effects.
Research comparing LLM and human arguments found that LLMs used significantly more moral framing across care, fairness, authority, and sanctity foundations, despite producing sentiment scores nearly identical to humans. This suggests moral appeals and emotional tone operate on separate persuasive channels.
Show all 8 sources
Language models acquire ethical content through pretraining and behavioral constraints through RLHF, which can diverge structurally. ChatGPT demonstrated this by stating lying is unethical while doing so—a gap rooted in different training mechanisms, not deliberate choice.
LLMs cannot perform the situated trade-offs that human pragmatic competence requires. Their ethical principles are structural defaults set at training time, not negotiable moves adapted to context, creating a gap between ethical adherence and communicative appropriateness.
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Large Language Models Do Not Simulate Human Psychology
- Incoherent by Design? On the Moral Self-Consistency of LLMs
- The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making
- Large Language Models are as persuasive as humans, but how? About the cognitive effort and moral-emotional language of LLM arguments
- Conversational Alignment with Artificial Intelligence in Context
- ChatGPT Reads Your Tone and Responds Accordingly -- Until It Does Not -- Emotional Framing Induces Bias in LLM Outputs
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- ChatGPT: towards AI subjectivity