When AI judges rank ideas head-to-head, can they tell you how good each one really is, or how sure to be?

This explores why an LLM acting as a judge can rank candidates against each other yet still fail to tell you how good each one really is, or how confident to be in that verdict. The corpus doesn't study tournament brackets directly. It does have a clear answer to the deeper problem: generating a good candidate and estimating its value are different skills, and LLMs are much stronger at the first. In scientific discovery settings, models reliably propose valid candidates in structured spaces but can't judge their true value or uncertainty. The fix that works is to hand that job to an external statistical model, a Gaussian process fitted to real experimental data, which can say both "this looks promising" and "we've barely explored this region" Can language models reliably judge their own candidate quality?. A tournament of LLM judgments has no real-world data like this anchoring it. It only compares the model's opinions with each other.

The judgments themselves are also coarse and easy to sway. When a judge answers with a single score or a single winner, many candidates end up tied, and the information about *how much* better one candidate is gets thrown away. Taking the expected value across the probabilities the model assigns to each possible score turns those discrete verdicts into continuous ones, and ties become far rarer Can reading logit distributions break ties in LLM judging?. The judge can also be fooled: LLM evaluators reliably give higher scores to answers that include fake references or polished formatting, whatever the substance Can LLM judges be tricked without accessing their internals?. In a tournament, that bias compounds. The candidate that looks most authoritative keeps winning rounds, so the ranking can reward presentation rather than quality.

Nor can you rerun the judge with temperature set to zero to get rid of the noise. Deterministic settings just repeat the same single draw from the model's distribution, so the output is consistent without being reliable Does setting temperature to zero actually make LLM outputs reliable?. A tournament run this way reports one sample's opinion as if it were the verdict, with no sign of how much a different draw would have changed the bracket.

The more hopeful thread is that the uncertainty signal often already exists inside the model. You just have to read it out instead of forcing a verdict. Token probabilities give well-calibrated uncertainty estimates that beat elaborate multi-call heuristics for deciding when to retrieve more information Can simple uncertainty estimates beat complex adaptive retrieval?. A model's confidence in its own answers can even stand in for an external verifier as a training signal Can model confidence alone replace external answer verification?. In personalized judging, asking the judge to state its own uncertainty and letting it abstain on hard cases pushes reliability above 80% on the cases it keeps Why do LLM judges fail at predicting sparse user preferences?. That suggests a tournament fails partly because of its format. Every match must have a winner, even when the judge has no real basis for choosing one.

The less obvious lesson is that "the LLM can't estimate uncertainty" and "the LLM's probabilities are well calibrated" are not contradictory. The failure usually comes from how the judgment is collected, through forced choices, single samples and discrete scores, and not from what the model knows. The same holds for LLM survey responses, where unrealistic answers disappear once you change how the answers are collected Why do LLMs give unrealistic survey responses?. If you want real value estimates, read the probability distribution instead of the winner, let the judge abstain, and where you can, ground the scores in outside evidence the model can't talk its way around.