Ask an AI to answer a survey in plain words, then convert that text to a 1–5 score. Can you trust the numbers?

This explores whether you can have a language model answer a survey in free text and then use embedding similarity to convert that text into a rating scale, and whether those numbers can be trusted. The corpus's direct answer is mostly yes for the measurement step, with a bigger caveat about who is being measured. One note describes Semantic Similarity Rating: prompt the model for a written answer, then map that text to the scale by comparing embeddings. It reaches about 90% of human test-retest reliability (how consistently real people answer when asked the same thing twice) and produces realistic response distributions Why do LLMs give unrealistic survey responses?.

The less obvious finding is why the mapping helps. When you ask a model to output a number directly, it tends to give oddly skewed, over-positive answers. Those problems largely vanish when the output channel changes to text plus embeddings. That suggests the strange survey behavior of LLMs was mostly an artifact of how we asked, not a limit of the model. A model that can say something nuanced in words can lose that nuance when forced to pick a digit, and the embedding step recovers it.

A reliable ruler still doesn't guarantee a representative respondent. Across 18 models and four datasets, aligned LLMs lean toward kinder, more socially desirable answers on value-laden questions. The bias grows with model size, traces to post-training alignment, and persists across prompt framings Do aligned language models consistently prefer kinder survey answers?. A separate analysis of 106 LLMs finds they cluster in a narrow region of value space while human respondents scatter widely Do large language models actually reflect human value diversity?. The notes don't reconcile this with the claim that over-positivity disappears. One plausible reading is that the embedding method fixes scale-use artifacts on things like product ratings, while alignment-driven bias on values questions is a different, deeper effect. That reading is an inference, not something the corpus tests.

Two more caveats. First, matching human distributions only helps if the human numbers meant something. Annotation research finds that responses mix genuine preferences, non-attitudes (people answering without holding a view), and constructed preferences (views made up on the spot), and that these need different handling Do all annotation responses measure the same underlying thing?. A model that reproduces human-looking ratings may be reproducing some of that noise. Second, models perform better on high-frequency phrasings than on equally meaningful rare paraphrases Do language models really understand meaning or just surface frequency?. It's worth asking whether free-text answers, and their embedding distances to the scale anchors, carry a similar sensitivity to surface wording. The corpus doesn't test this directly.

The takeaway is that embedding similarity looks like a much better way to read a model's answer than forcing it to output a number. It doesn't show that the model answers like a diverse human population, and the corpus's evidence on that point runs the other way.