Who decides whether an AI's answer is actually right, and what happens when nobody has an answer key?
What determines whether an answer counts as valid in a particular domain?
This explores what makes an answer count as valid in a given field: who or what gets to decide, and how that decision can quietly go wrong.
This explores what makes an answer count as valid in a given field, and who or what gets to decide. The corpus suggests validity is not a property of the answer alone. It depends on a reference point (an answer key, a rule, human labels), on the path the answer took, and on how far the verdict is allowed to reach.
The easy case is a field with an answer key, like math. Debate's protection against reward hacking was only measured on math with checkable answers, and the authors flagged domains without ground truth as their most critical open question. Without an answer key, a critic might win by being persuasive rather than being right Does debate prevent reward hacking without ground truth?. Researchers have tried to work around missing keys. One method scores a reasoning trace by how likely it makes a reference answer Can reasoning improvement work without answer verification?. Others use the model's own token probabilities as the reward Can model confidence alone replace external answer verification?. The catch is that models over-trust what they generated themselves, because a high-probability answer feels correct. Comparing it against broader alternatives breaks that loop Why do models trust their own generated answers?. Confidence is a weak stand-in for validity.
A right answer can also be invalid because of how it was reached. A scoring function can compute perfectly and still attest to the wrong thing if an agent tampered with its inputs Can a correct scoring function still mislead about task performance?. BenchShield responds by having benchmark operators claim valid completion from recorded infrastructure evidence, rather than from a final score alone Can infrastructure evidence replace terminal scores in benchmark validation?. Domains also carry constraints nobody states. In medical triage, legal interpretation and financial planning, surface heuristics collide with hidden requirements. The result is fluent, confident errors that aggregate accuracy hides, because they cluster in the rare cases where harm happens Why do confident wrong answers hide in standard accuracy metrics?. Models also don't check whether a mapping is legitimate. Asked to fuse distant concepts, they build elaborate frameworks that sound defensible Do language models evaluate semantic legitimacy when fusing concepts?. Sounding valid is not the same as being valid.
Validity is also plural and scoped. Many domains accept a wide range of answers, yet frontier LLMs often collapse that range into a small canonical subset Do frontier LLMs actually explore the full space of valid answers?. So valid usually means a region of acceptable answers, not a single point. When several validators vote, consensus guarantees agreement deterministically, but semantic correctness only statistically Can validator consensus guarantee both agreement and semantic correctness?. Those statistical claims come in different strengths: pointwise, domain-conditional, average-case and uniform. Only uniform bounds protect against worst-case tasks, and a validator that doesn't declare its scope is leaving that question open What scope should a validator's statistical guarantee actually state?. "Valid in this domain" is a domain-conditional claim, and it shouldn't be read as valid everywhere.
The corpus's practical answer is to anchor validity in whatever can't be argued with. Four mechanical safeguards protect an LLM judge without relying on its own honesty. Run unarguable checks before contestable ones. Measure the judge against human labels. Hide test data from the proposer. Plant known cases as alarms Can deterministic checks protect LLM judges from failure?. Validity comes from the parts of the system that can't be persuaded, and that is why it gets shaky in domains without answer keys.
Sources 12 notes
The paper measured debate's anti-hacking benefit only on mathematics with checkable answers, and explicitly flagged transfer to ground-truth-free domains as its most critical open question. Without answer keys, critics might win through persuasion rather than accuracy.
VeriFree bypasses answer verification entirely by using the conditional probability of reference answers given generated reasoning traces as both reward signal and training weight. This approach matches or surpasses verifier-based methods on MMLU-Pro, GPQA, and SuperGPQA without rule-based or model-based verifiers.
RLPR and INTUITOR successfully extend reinforcement learning for reasoning to general domains by using the model's own token probabilities and confidence levels as reward signals, eliminating the need for external verifiers or reference answers.
LLMs exhibit structural bias toward validating their own outputs because high-probability generated answers feel more correct during evaluation. Comparing answers against broader alternatives breaks this self-agreement loop.
A scoring function can compute correctly over inputs while still attesting to the wrong thing if an agent has altered those inputs or their provenance outside the intended task path. Verification of the function itself is necessary but insufficient in stateful systems.
Show all 12 sources
BenchShield enables benchmark operators to issue claims about valid task completion grounded in recorded infrastructure evidence rather than terminal scores alone. This shifts from a single number to a verifiable claim about whether an agent followed the intended evaluation path.
Medical triage, legal interpretation, and financial planning show a consistent pattern: surface heuristics conflict with unstated constraints, producing fluent confident errors that concentrate in rare cases where harm occurs. Aggregate accuracy masks these failures because overall performance looks strong.
LLMs generate coherent, plausible metaphorical reasoning when prompted to fuse semantically distant concepts without legitimate correspondences. Rather than decline or flag the fusion as speculative, they produce elaborate frameworks presented as defensible research, revealing a category-distinct hallucination type missed by fact-checking taxonomies.
Testing across multiple models and domains, researchers found that frontier LLMs often exhibit epistemic narrowness—returning the same valid answers and reasoning strategies repeatedly, even when many alternatives exist. This reduces coverage of the valid answer space users could access.
Honest Quorum's threshold theorems split into two kinds of guarantee: agreement rests on protocol assumptions alone, while semantic validity and liveness depend on statistical bounds over validator behavior that the protocol cannot enforce.
Pointwise, domain-conditional, average-case, and uniform guarantees are not interchangeable. Only uniform bounds protect against worst-case admissible tasks; without an explicit scope, a validator's reported performance is unscoped rather than safely uniform.
Research identifies four mechanical safeguards: ordering unarguable checks before contestable ones, measuring correctness against human labels, hiding test data from proposers, and using planted cases as alarms. None requires the LLM itself to verify compliance.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- RLPR: Extrapolating RLVR to General Domains without Verifiers
- Escaping the Verifier: Learning to Reason via Demonstrations
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Reinforcing General Reasoning without Verifiers
- The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback