When a data label is called 'adjudicated,' does that mean people agreed on it, or that it's actually correct?

This explores whether an 'adjudicated' label means the labelers converged on the same answer or that the label was checked against an outside standard, and whether the two can come apart. The corpus has no note that defines adjudication directly, so treat what follows as the closest conceptual neighbors. They agree on one point: agreement and correctness are different guarantees, and a label can have one without the other. In the Honest Quorum work on validator panels, agreement follows from the protocol's own rules, but whether the agreed answer is semantically valid depends on statistical assumptions about validator behavior that the protocol cannot enforce (Can validator consensus guarantee both agreement and semantic correctness?). If adjudication means a panel settled on one answer, you are guaranteed one answer, not the right one.

Agreement can also be hollow. The same paper lists eight things validators can share, from model weights to provider infrastructure, so one shared fault can make several 'independent' validators agree for the same wrong reason (Can a quorum of validators really provide independent judgment?). Those are AI validators, not human annotators. But the parallel is easy to see: labelers working from the same guidelines, tool, and background may converge because of a common cause. The reverse also holds. Matching the ground truth does not prove the labeler did the work, because agents that skipped a required verification step still produced verdicts matching ground truth (Can a correct outcome hide protocol violations in multi-agent systems?). So neither route, agreeing with each other or matching a standard, tells you how the label was reached.

Sometimes there is no single standard to match. For socially embedded sentences, readers from different positions read them differently, and that disagreement reflects valid differences in perspective, not annotation failure (Why do readers interpret the same sentence so differently?). Annotation responses also fall into three kinds: genuine preferences, non-attitudes, and constructed preferences. Two annotators agreeing could be two non-attitudes happening to coincide, and treating all three kinds alike contaminates downstream training (Do all annotation responses measure the same underlying thing?).

So when you see 'adjudicated,' ask two questions. Was the resolution independent, or did the resolvers share the same blind spots? And does the item have a standard at all, or is it contested ground? Where it is contested, forcing consensus can erase useful information. Disagreement in reasoning traces can mark legitimately contested value territory that should be escalated, not voted away (Can disagreement in reasoning traces signal legitimate value conflicts?). A related idea is dialectical reconciliation, where both sides adjust until their positions are compatible but not identical. Current AI systems tend to collapse that into false agreement or one side winning (Can disagreement be resolved without either party fully yielding?). An adjudicated label may therefore be a flattened compromise, not a verified truth.