When AI agents disagree, should we vote the split away or read it as a signal of where the answer is shaky?

This explores what a multi-agent system should do when independent runs disagree. The usual instinct is to collapse the split into one answer by majority vote or a consensus round. The corpus suggests that instinct often throws away the most useful thing the system produced. The shape of the disagreement can tell you more than the winning answer does.

Start with the evidence that disagreement carries signal. In multi-agent coding of qualitative data, accuracy was higher when the agents argued for a long time and left labels unresolved. How often they disagreed turned out to be a reliable sign of quality, not a sign of failure Does disagreement between AI coders signal better accuracy?. A related line of work separates two kinds of disagreement. In one, agents reason from the same facts and still reach different conclusions. That pattern marks territory where reasonable values conflict. Voting erases exactly that, even though those are the cases a human should probably decide rather than leave to automation Can disagreement in reasoning traces signal legitimate value conflicts?. Seen this way, aggregation is partly a routing decision: some splits should be resolved, and others should be escalated.

The opposite failure is agreement that comes too easily. Multi-agent reasoning systems reached consensus without any real disagreement 61% of the time. The authors trace this to training that rewards models for accommodating rather than challenging, and the same pressure leads a single model revising its own work to grow more confident in wrong answers Why do AI systems agree when they should disagree?. Groups of agents also tend to accept what their neighbours say without checking it, which lets errors spread through the network Why do multi-agent systems fail to coordinate at scale?. One model even predicts how agent communities change their opinions by assuming each agent moves toward whatever position brings the least social pressure Can we predict how agent communities shift opinions?. If social pressure can predict where a group ends up, then the group's consensus doesn't count as independent evidence. Agreement reached through discussion is weaker evidence than agreement among runs that never saw each other.

Practical designs try to steer between these two failures. One adds a dedicated agent whose only job is to judge whether the others have actually agreed. This prevents both endless stalling and premature convergence, and it reached outcomes comparable to real human decision conferences Can AI systems detect when they've genuinely reached agreement?. That matters because LLM groups trying to reach agreement usually fail by stalling and timing out, not by quietly settling on a wrong value, and the problem gets worse as groups grow Can LLM agent groups reliably reach consensus together?. Other approaches weight agents instead of counting votes. Contribution scoring can switch off agents that add little during a run Can multi-agent teams automatically remove their weakest members?. An agent acting as judge, gathering its own evidence before ruling, was dramatically more stable than a plain LLM judge, though errors in its memory module still cascaded Can agents evaluate AI outputs more reliably than language models?. There is also a third outcome besides one side winning or false agreement. In dialectical reconciliation, both positions shift until they are compatible but not identical, and current AI systems rarely do this Can disagreement be resolved without either party fully yielding?.

One caution before investing in elaborate aggregation: about 80% of the variation in multi-agent performance tracks how many tokens were spent, not how cleverly the agents coordinated How does test-time scaling work at the agent level?. Some of what looks like the benefit of combining runs may just be the benefit of computing more. Taken together, a good design would keep runs independent, record where and why they split, use a judge that checks evidence to settle factual splits, and send splits about values to a human. The corpus doesn't yet have a head-to-head test of these aggregation strategies, so that recipe is pieced together from separate studies rather than taken from one benchmark.