When AI agents debate and trust whoever sounds most sure, are they tapping real expertise, or just rewarding confidence?
Does confidence-based weighting in deliberation substitute for competence-based expertise?
This explores whether AI groups that weigh each member's confidence when deliberating end up doing the job that real expertise does in human groups, or whether confidence is only a stand-in for competence.
This explores whether confidence-weighted deliberation among AI agents can stand in for competence-based expertise. The corpus says it can't: confidence is an observable signal, and competence is a hidden fact, and the two can come apart. One note describes multi-agent deliberation as a mixture-of-experts system whose routing follows what agents *sound* sure about, not what they're actually good at. When confidence is miscalibrated, the group can manufacture a confident consensus even when a dissenting agent holds better evidence Does confidence drive influence in multi-agent deliberation systems?.
Confidence is a fair signal in some settings. Models that are more confident are measurably steadier: they hold up under prompt rephrasing, and larger models and objective tasks show higher confidence and more robustness Does model confidence predict robustness to prompt changes?. Researchers also use confidence variance as a diagnostic to steer reasoning away from overthinking or underthinking Can confidence patterns reveal overthinking versus underthinking?. Confidence says something real about a model's internal state, which is why it makes a tempting weighting scheme. But being stable is not the same as being right, and a model can be stably wrong.
Training makes the gap wider. RLHF rewards confident-sounding answers over clarifying questions, and models end up with 77.5% fewer grounding acts than humans use Does preference optimization harm conversational understanding?. The agents in a deliberation have therefore been tuned to sound sure, which weakens confidence as a competence signal. Confidence can also be repaired. Using answer-span confidence as a training reward restores calibration that RLHF eroded Can model confidence work as a reward signal for reasoning?. That suggests confidence-weighting may work better once the confidence has been calibrated, but the corpus doesn't test that in deliberation itself.
Human expertise works differently. Human debates get settled by argument quality, social authority, cultural context and trust. Multi-agent LLM debates run on probability ranking, and that gap leads AI systems to amplify errors in contested domains where expertise matters most How do LLM debates differ from human expert consensus?. LLMs also lose the reputation and track record that give an expert's claim its force, because they see only text Can language models distinguish expert arguments from common assumptions?. Nothing in a text-only deliberation records who has been right before, so confidence fills the empty slot.
The evidence that expertise can't be swapped out comes from ideation. Diverse agent teams beat solo work only when members have real senior domain knowledge. Diverse but non-expert teams do worse than one competent agent, because stimulation without knowledge produces process losses instead of insight Does cognitive diversity alone improve multi-agent ideation quality?. So the answer to the question is no. Confidence can be a useful input, but it works as a proxy for competence, and it fails exactly when the agents are confidently wrong.
Sources 8 notes
Multi-agent LLM deliberation works like a mixture-of-experts system, but adaptive routing keys off observable confidence signals rather than actual task competence. This means miscalibrated confidence manufactures misleading consensus even when agents disagree with better evidence.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
ReBalance uses confidence variance and overconfidence as diagnostic signals to apply training-free steering vectors that reduce overthinking redundancy while promoting exploration during underthinking, improving accuracy across models from 0.5B to 32B parameters.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Show all 8 sources
Multi-agent LLM debates operate through chain-of-thought probability ranking, fundamentally different from human debates which are settled by argument quality, social authority, cultural context, and interpersonal trust. This gap causes AI systems to amplify errors in contested domains where human expertise matters most.
LLMs lose the social context that gives expert claims their force—reputation, track record, and standing—because they process only text, not the social world where expertise is built and evaluated.
Multi-agent teams substantially outperform solo ideation, but only when members possess genuine senior knowledge. Diverse teams without expertise underperform even a single competent agent, because cognitive stimulation without expertise triggers process losses instead of insight.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Debating with More Persuasive LLMs Leads to More Truthful Answers
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Reported Confidence in LLMs Tracks Commitment More Than Correctness
- The Thin Line Between Comprehension and Persuasion in LLMs
- Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments
- Finding Common Ground: Using Large Language Models to Detect Agreement in Multi-Agent Decision Conferences