If several AI checkers see the same prompt and evidence, does their majority vote confirm anything — or just echo one input?
How much do shared prompts and evidence channels correlate validator outputs?
This explores whether validators that get the same prompt and the same evidence tend to make the same mistakes together, so that a majority vote looks like independent confirmation when it is really one input counted several times.
This explores whether validators that get the same prompt and the same evidence tend to make the same mistakes together, so that a majority vote looks like independent confirmation when it is really one input counted several times. The corpus has no single number for "how much." It does show that the correlation is real and structural, and it points to where the effect is likely to be largest.
The most direct evidence is the Honest Quorum work. It lists eight things agentic validators can share, ranging from model weights to provider infrastructure. One fault in a shared piece can then hit many validators at once, and quorum voting reads the resulting agreement as independent confirmation when it may just be a common cause Can a quorum of validators really provide independent judgment?. The same paper's theorems show why this matters. Agreement holds from the protocol alone, but semantic correctness depends on statistical bounds over validator behavior that the protocol cannot enforce Can validator consensus guarantee both agreement and semantic correctness?. Those bounds only work if validators err independently, and shared inputs undermine that. The notes describe the mechanism but do not measure how much correlation results.
Other notes suggest why a shared prompt is a strong shared lever. Prompt wording can move outputs a lot: low-confidence models swing widely under rephrasing, while confident ones hold steady Does model confidence predict robustness to prompt changes?. The effect also depends on the model. Step-by-step prompting helps cheaper models and can hurt stronger ones Do prompt techniques work the same across all LLM tiers?. So validators of the same tier plausibly respond to a shared prompt in the same way, while a mixed panel would not. Chain-of-thought exemplars with invalid logic work almost as well as valid ones, so the prompt's form drives behavior more than its content Does logical validity actually drive chain-of-thought gains?. Validators sharing a template therefore inherit the same quirks. Judges also share exploitable blind spots. Fake references and rich formatting fool them without any access to the model, and the authority and beauty biases ignore the meaning of the text Can LLM judges be fooled by fake credentials and formatting?. A single doctored piece of shared evidence could fool every validator that has that bias.
The evidence channel may matter even more than the prompt. Agent-based evaluation that collects its own evidence cut judge shift to 0.27%, against 31% for an LLM judge working from what it was handed Can agents evaluate AI outputs more reliably than language models?. That figure measures instability, not correlation between validators, but it suggests evidence handling is where outputs are won or lost. The same study found that a shared memory module cascaded errors, which is a concrete case of one channel carrying a fault to everything downstream. One way to break the link is to ground claims in evidence that does not pass through the judging model. BenchShield does this for benchmarks, using recorded infrastructure evidence instead of terminal scores Can infrastructure evidence replace terminal scores in benchmark validation?.
The practical upshot is that agreement among validators tells you less than its count suggests. A related finding points the same way: filtering reasoning traces by step-level quality matched the gains of majority voting with far fewer traces, so quality mattered more than quantity Does step-level confidence outperform global averaging for trace filtering?. Adding more validators that share a prompt, model lineage and evidence source adds little independent information. Diversifying those inputs, or having validators gather their own evidence, is where the corpus points, and none of the notes here quantifies the payoff.
Sources 9 notes
The Honest Quorum paper identifies eight things agentic validators may share—from model weights to provider infrastructure—allowing one fault to affect multiple validators simultaneously. Quorum voting then counts agreement as independent confirmation when it may simply reflect a common cause.
Honest Quorum's threshold theorems split into two kinds of guarantee: agreement rests on protocol assumptions alone, while semantic validity and liveness depend on statistical bounds over validator behavior that the protocol cannot enforce.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
A 23-prompt benchmark across 12 LLMs shows rephrasing and background-knowledge prompts boost cheap models, while step-by-step reasoning reduces accuracy in high-performance models. Task structure, not generic best practices, determines which prompts help.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
Show all 9 sources
Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
BenchShield enables benchmark operators to issue claims about valid task completion grounded in recorded infrastructure evidence rather than terminal scores alone. This shifts from a single number to a verifiable claim about whether an agent followed the intended evaluation path.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting
- The Honest Quorum Problem: Epistemic Byzantine Fault Tolerance for Agentic Infrastructure
- When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
- Deep Think with Confidence
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Can AI Agents Agree?