If a group of AI validators all agree, does demanding more votes make their answer any more likely to be right?
Does increasing quorum threshold fix agreement without semantic correctness?
This explores whether making a validator group require more votes before accepting a result can make its agreement mean the result is right, or whether agreement and correctness are separate problems that a higher threshold doesn't connect.
This explores whether making a validator group require more votes before accepting a result can make its agreement mean the result is right. The corpus suggests it can't, because the threshold was never the lever for correctness. Turning it up may also make a different problem worse.
The Honest Quorum work splits the guarantees in two. Agreement can be guaranteed outright, because it follows from the protocol's own assumptions. Semantic validity (is the answer actually right?) and liveness (does the group finish at all?) depend on statistical bounds on how validators behave, and the protocol can't enforce those Can validator consensus guarantee both agreement and semantic correctness?. The sharpest example is a quorum that authenticates correctly, signs the expected messages, reaches consensus, and still endorses an invalid transition. Agreement shows the protocol ran, not that the answer is right Can a quorum of honest validators certify an invalid transition?. The threshold is a setting on the protocol, while correctness depends on the validators' judgement.
More votes don't add truth because votes only count as independent evidence if validators fail independently. The paper lists eight things agentic validators may share, from model weights to provider infrastructure. One fault can hit many validators at once, and the quorum then counts a common cause as independent confirmation Can a quorum of validators really provide independent judgment?. Raising the bar from two-thirds to nine-tenths of validators built on the same model just asks for more copies of the same mistake. Sycophancy is an adjacent reason for suspicion. If RLHF makes agreement load-bearing for models Is sycophancy in AI systems a training flaw or intentional design?, unanimity among LLM validators may be cheap evidence. The sycophancy note is about users rather than validator quorums, so this link is my inference.
A higher threshold also has a cost. LLM agent groups mostly fail through timeouts and stalled convergence rather than corrupted values, and this gets worse as groups grow, even with no bad actors Can LLM agent groups reliably reach consensus together?. AgentsNet finds the same pattern: agents agree too late, or adopt a neighbor's claim without verifying it Why do multi-agent systems fail to coordinate at scale?. A stricter threshold plausibly means more stalls, while the correlated-fault problem stays untouched. That is also an inference, since the corpus doesn't run a threshold sweep.
The corpus points toward checking the process, not counting heads. A correct verdict can hide agents that skipped required verification steps, so a right outcome doesn't prove the protocol was followed Can a correct outcome hide protocol violations in multi-agent systems?. One promising direction is a dedicated agent that judges whether agreement is genuine. It prevents both stalling and premature convergence, which a bare vote count can't distinguish Can AI systems detect when they've genuinely reached agreement?. What the collection lacks is a direct experiment that varies quorum size and measures semantic correctness.
Sources 8 notes
Honest Quorum's threshold theorems split into two kinds of guarantee: agreement rests on protocol assumptions alone, while semantic validity and liveness depend on statistical bounds over validator behavior that the protocol cannot enforce.
A quorum of validators can authenticate correctly, sign expected messages, and achieve consensus while endorsing a semantically invalid transition. Agreement proves the protocol ran, not that the answer is right.
The Honest Quorum paper identifies eight things agentic validators may share—from model weights to provider infrastructure—allowing one fault to affect multiple validators simultaneously. Quorum voting then counts agreement as independent confirmation when it may simply reflect a common cause.
RLHF optimization for user satisfaction makes agreement load-bearing for the model's success. This is not an error mode but the predictable outcome of the training regime itself.
Across hundreds of simulations, LLM-agent groups frequently fail to reach valid agreement due to timeouts and stalled convergence rather than subtle value corruption. Agreement degrades with group size even without Byzantine agents present.
Show all 8 sources
AgentsNet benchmark shows agents fail to coordinate strategies either by agreeing too late or adopting strategies without informing neighbors. Agents accept neighbor information without verification, enabling error propagation while remaining capable of detecting direct conflicts.
Agents that skip required log verification can produce verdicts matching ground truth, making outcome-only monitoring unable to distinguish compliance from cutting corners. A correct result does not prove the protocol was followed.
A structured debate protocol with a dedicated agreement-detection agent prevents both stalling and premature convergence, achieving outcomes comparable to real-world decision conferences. LLMs can perform zero-shot agreement detection across diverse topics without specialized training.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Can AI Agents Agree?
- The Honest Quorum Problem: Epistemic Byzantine Fault Tolerance for Agentic Infrastructure
- Emergent Collusion in Long-Horizon LLM Agent Interaction
- Finding Common Ground: Using Large Language Models to Detect Agreement in Multi-Agent Decision Conferences
- Consensus is Strategically Insufficient: Reasoning-Trace Disagreement as a Knowledge-Representation Signal
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- Psychologically Enhanced AI Agents