If you test an AI once per scenario, are you seeing how it behaves or just one lucky roll of the dice?
What happens when we use a single response per condition?
This explores what goes wrong when a model's behavior (or a person's judgment) is measured from one sampled answer per test condition instead of several, and why one draw is a shaky unit of evidence.
This explores what goes wrong when you judge a model, or a human annotator, from one sampled answer per condition. No note in the collection tests this head-on, so this is stitched together from neighboring material. It all points the same way: one response is a single draw from a spread of possibilities, and it can't tell you how wide that spread is.
The clearest picture comes from how LLMs behave. Shanahan's 20-questions test asks the model the same thing again, and each regenerated answer is different yet consistent with everything said so far. That means the model never committed to one character (Do large language models actually commit to a single character?). A companion note describes the model as holding a superposition of many possible characters that only narrows as the conversation goes on (Does an LLM commit to a single character or maintain many?). Suppose your two conditions produce different answers on a single draw each. You can't tell whether the condition mattered or the dice landed differently. The problem is worst early in a conversation, when the spread is widest.
Human responses have the same problem. Behavioral science suggests an annotation can be a genuine preference, an answer given with no real opinion behind it, or a preference constructed on the spot. What tells them apart is consistency across measurement conditions (Do all annotation responses measure the same underlying thing?). One response per condition contains no consistency information, so all three look identical. Treating them the same then contaminates reward model training. For both machines and people, the thing you want is the pattern across repeats, and a single observation has no pattern in it.
Single-response checks also hide what optimization does to a model. RL that rewards only final-answer correctness sharpens the policy and shrinks its diversity, and that loss spills over to problems it hasn't solved (Does outcome-based RL diversity loss spread across unsolved problems?). Diversity only exists across many responses, so one sample per problem can look fine while the model's range narrows. Prompts have a related trap. Prompts tuned without knowing the inference strategy (best-of-N, majority voting) systematically underperform, and tuning both together gave up to 50% improvement (Does prompt optimization without inference strategy fail?). A one-response test measures how good a prompt is at one draw, which is a different question from how it performs when sampled repeatedly.
The last cost is overclaiming. Statistical guarantees come in different scopes (pointwise, domain-conditional, average-case, uniform), and they aren't interchangeable. Without a stated scope, a result is unscoped rather than safely general (What scope should a validator's statistical guarantee actually state?). One response per condition supports at most a pointwise claim, and it usually gets reported as something broader. The design side has the same theme. Deterministic single-path reasoners can't represent problems with several valid answers, while stochastic ones can hold a distribution and branch (Can stochastic latent reasoning let models explore multiple solutions?). The takeaway is to stop asking what the model said and ask what it tends to say and how much that varies.
Sources 7 notes
Shanahan's 20-questions test shows LLMs maintain a superposition of consistent objects or characters and sample from that distribution at generation time. Regenerating the same response yields different outputs, each consistent with prior context, proving no fixed commitment exists.
Research shows LLMs don't commit to a single character but instead maintain a probability distribution over many consistent simulacra. Each response samples from this distribution, explaining why regenerations can yield different personalities while remaining consistent with prior context.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
RL that rewards only final answer correctness sharpens the policy globally, concentrating probability mass on correct trajectories for solved problems while simultaneously reducing diversity on unsolved ones. Historical exploration (training diversity via UCB-style bonuses) and batch exploration (test-time diversity via repetition penalties) require structurally different mechanisms.
Prompts optimized without knowledge of the inference strategy (best-of-N, majority voting) systematically underperform. Joint optimization of both prompt and inference strategy yields up to 50% improvement across reasoning and generation tasks.
Show all 7 sources
Pointwise, domain-conditional, average-case, and uniform guarantees are not interchangeable. Only uniform bounds protect against worst-case admissible tasks; without an explicit scope, a validator's reported performance is unscoped rather than safely uniform.
GRAM replaces deterministic latent updates with stochastic sampling, enabling models to represent probability distributions over solutions rather than single points. This lets recursive reasoners maintain uncertainty, explore alternatives, and handle ambiguous or multi-solution problems that deterministic single-path designs cannot.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- Measuring Human Preferences in RLHF is a Social Science Problem
- Outcome-based Exploration for LLM Reasoning
- Vector Policy Optimization: Training for Diversity Improves Test-Time Search
- Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training
- Jointly Reinforcing Diversity and Quality in Language Model Generations