Chatbots are trained to be helpful and safe, so when one pretends to be a survey respondent, does it turn too nice?

This explores why the training step that makes chatbots helpful and safe tilts the answers when an LLM plays a survey respondent. The corpus's most direct finding is that the tilt is toward kindness. Across 18 models and four datasets, aligned models lean toward safer, more socially desirable answers on value-laden questions. The lean gets stronger as models get bigger, traces back to post-training, and survives rewording the prompt Do aligned language models consistently prefer kinder survey answers?. The result is a simulated crowd that is nicer than the real one, and it can't authentically reproduce the perspectives of people who hold harsher or less agreeable views.

The likely mechanism is that alignment trains a model to be the answerer that human raters prefer, not the answerer that matches the spread of real opinion. That makes it a chain of human design choices. One note finds that RLHF and DPO create measurable gaps across English dialects and global opinions, and it ties them to who the annotators were and how the task was defined, not to anything inevitable How does LLM alignment affect representation across dialects?. Post-training also seems to work by surfacing a narrow slice of what pretraining already contains, since about 1,000 curated examples can be enough Can careful curation replace massive alignment datasets?. So when you ask an aligned model to be a survey respondent, you may be getting the polite assistant persona in a costume, not a sample of humanity.

Two other notes show that alignment narrows output in general. RL training amplifies one dominant format from pretraining within the first epoch and collapses the alternatives Does RL training collapse format diversity in pretrained models?. That note measures formats, not opinions, but it shows the same squeezing pattern. Different LLMs also independently converge on strikingly similar answers, and the note blames overlapping training data and alignment procedures Do different AI models actually produce diverse outputs?. That means swapping in a different model won't restore the diversity. A reward for kind, safe answers is also a stand-in for what you actually want, and one note argues that optimizing against a signal that only partly captures the task is the shared root of reward hacking Does reward hacking always stem from the same failure?. Read that way, benevolence bias is the model doing well at a proxy goal.

The twist is that alignment may not be the whole story. When a model is asked to write a short text answer, and an embedding step then maps that text onto the rating scale, it reaches about 90% of human test-retest reliability with realistic response distributions. The pathological skew and over-positivity disappear, which the authors read as an artifact of how the answer was extracted, not a limit of the model Why do LLMs give unrealistic survey responses?. The two findings can both hold. The benevolence note says rewording the prompt doesn't help, while this one changes the output channel, from a picked number to free text. So the polite-assistant prior may hit hardest when the model has to commit to a scale point, and alignment may have shaped the model without permanently distorting what it knows about people. The corpus doesn't test this directly, so it remains an open question.