When one AI team beats another, did the design win — or did it just spend more?

This explores why different multi-agent designs (team structures, sizes, coordination schemes) have to be measured on the same yardsticks before anyone can say one beats another. The corpus suggests that without shared measures, many apparent wins are illusions. You end up comparing budgets, or comparing a system to a baseline that was never measured the same way, instead of comparing designs.

Cost is the clearest case. One line of research finds that roughly 80% of multi-agent performance variance comes from token budget, not coordination cleverness How does test-time scaling work at the agent level?. A team that beats another while spending three times the tokens may just be spending more. Reporting cost next to accuracy is also what makes a real efficiency gain visible, such as shared caches that decouple performance from token spend. The same reasoning applies to the finding that small language models can handle most agent subtasks at 10-30x lower cost Can small language models handle most agent tasks?. That gap only matters if cost is a reported number and not an afterthought.

Success rate alone also hides too much. Two agent systems with identical task success can differ enormously in efficiency, reliability, and memory and verification overhead, which is why some researchers call for harness-level benchmarks that track the whole trajectory How should we measure agent system performance beyond task success?. That connects to the claim that reliability comes from the memory, skills, and protocols wrapped around the model, not the model alone Where does agent reliability actually come from?. Comparing designs really means comparing harnesses. There is also a subtler trap: putting several agents in a room doesn't prove a multi-agent effect. Interaction can leave a failure unchanged, amplify it, create it through composition, or define a new property, and only the last three are genuinely multi-agent Does a multi-agent setting automatically signal a security effect?. You can't tell which one you're seeing without a single-agent baseline measured the same way.

Once the numbers line up, patterns emerge that no single paper could see. Across 180 configurations, coordination stopped helping once a single agent already scored above about 45% accuracy, and topology choice alone swung error amplification by 4-17x When does adding more agents actually help systems?. Holding the task fixed while growing the network shows coordination breaking down at scale, through agreeing too late or accepting neighbors' claims unchecked Why do multi-agent systems fail to coordinate at scale?. Consensus studies show that even benign LLM groups fail mostly by stalling, not by absorbing corrupted values Can LLM agent groups reliably reach consensus together?. That distinction between losing liveness and corrupting values only exists if you measure the two separately.

The same idea works at the level of individual agents. Contribution scoring gives each agent a number, so a team can drop its weakest members at inference time Can multi-agent teams automatically remove their weakest members?. Capability vectors put agents in a shared space where they can be matched against budget and policy constraints Can semantic capability vectors replace manual agent routing?. Both only work when scores mean the same thing from one agent to the next. So the question stops being whether more agents help. It becomes what they help with, at what cost, and measured against what.