Can agreement across samples reveal when models are wrong?

Alongside retrieval, the authors add an internal check they call the Consistency Veto, defined as 1 − Pint. For each query they draw K = 5 stochastic samples and compute ρconsistent, the fraction of unordered sample pairs whose NLI contradiction probability stays below 0.5 in both directions. Pint = 1 − ρconsistent, so stable generations sit near 0 and mutually inconsistent ones approach 1. As pairwise inconsistency rises, the gate lowers the score "regardless of external evidence." The abstract reports, in the authors' word "unexpectedly," that this gate "carries most of the discriminative signal on dynamic queries." The excerpt omits the audit's results section and does not reproduce Equation 1, so how the gate enters the final score is known only from this description.

The authors state the limit directly. The gate "can suppress inconsistent confabulations, but it cannot detect false answers that recur consistently across samples." A model that gives the same wrong answer five times passes it. In the discussion, the gate suppresses scores on an ambiguous query (the Indonesia capital transition), which the authors call "a partial safety rail for this offloading, not a guarantee." They add that "high-density misconceptions can still evade the veto," so a confidently repeated error can carry a high score.

The gate is also set against prior work. The authors say it is "inspired by semantic entropy" but "neither forms semantic-equivalence clusters nor weights them by generation probabilities." It is therefore a pairwise agreement heuristic, not a semantic-entropy estimate. This connects to Should we treat LLM outputs as real empirical data?. Both read the spread of sampled outputs as information about belief rather than observation. The gate turns that spread into a trust parameter, which works when the model is unstable and fails when it is consistently wrong, because agreement then looks like confidence.

The excerpt gives no precision, recall or false-alarm rate for the gate, and no per-query-type breakdown of the audit beyond the abstract's phrasing. The dynamic-query finding therefore rests on the abstract alone, and the excerpt does not test whether it depends on the composite dataset or on K = 5. The implication is narrower than the finding's emphasis suggests. Any system that reads agreement across samples as evidence inherits a blind spot for systematic error. The gate can be a partial safeguard against inconsistency, but the excerpt gives no basis for treating it as a truth check.