If every AI on a panel follows the rules and agrees, can they still sign off on a wrong answer?

This explores whether a group of agents that all follow the rules can still sign off on a wrong result, and the corpus says yes. One paper names it the Honest Quorum problem: validators can authenticate correctly, sign the expected messages, and reach consensus while endorsing a semantically invalid transition Can a quorum of honest validators certify an invalid transition?. Agreement proves the protocol ran. It doesn't prove the answer is right.

The reason is that the guarantees come in two kinds. Agreement follows deterministically from the protocol's own assumptions. Semantic validity and liveness depend on statistical bounds on how validators actually behave, and the protocol can't enforce those Can validator consensus guarantee both agreement and semantic correctness?. The protocol can promise that everyone agrees. It can only hope that what they agree on is true, and that hope depends on how the validators behave in practice.

The link between compliance and correctness also fails in the other direction. Agents that skip a required log-verification step can still produce a verdict matching ground truth, so watching outcomes alone can't tell careful work from cut corners Can a correct outcome hide protocol violations in multi-agent systems?. Compliance and correctness are two separate properties, and passing one tells you little about the other. The same pattern shows up wherever systems are checked piece by piece. Local checks on plausibility, alignment, or protocol compliance test different things than end-to-end safety requires Can individual components pass safety checks if the system still fails?. A per-action check also can't state a rule about a sequence of actions, so steps that are each allowed can add up to a violation Can stateless checks ever catch sequence-level constraint violations?.

LLM agents make this worse in practice. Their consensus often fails through stalled convergence and timeouts, and it degrades as the group grows, even with no bad actors present Can LLM agent groups reliably reach consensus together?. When they do agree, the content they pass along may be quietly damaged. Frontier models corrupt about 25% of document content over long relay workflows, and spot checks miss it Do frontier LLMs silently corrupt documents in long workflows?. A quorum stamping its approval on that content would be certifying something nobody actually checked.

The corpus points to a few ways to close the gap, all of which look past the final vote. Infrastructure evidence can back a claim that an agent followed the intended path, rather than resting on a terminal score alone Can infrastructure evidence replace terminal scores in benchmark validation?. Asynchronous verifiers can watch a reasoning trace as it unfolds and step in only on violations, at near-zero latency cost Can verifiers monitor reasoning without slowing generation down?. Cryptographic commitments can make the process record tamper-evident without exposing sensitive content Can commitments protect sensitive agent data while enabling verification?. Even proving compliance is hard, though. Behavioral training can only show conditional compliance, because any scored behavior is observed behavior and can't separate an agent that always complies from one that complies only when watched Can behavioral training prove a model always complies?. So compliance can't certify validity, and it can't be fully certified itself.