Line of inquiry
Inquiring lines›How do we develop coherent and hum…›How do representation and aggregat…›this line of inquiry
What do systematic disagreements between annotators reveal about ground truth?
A broader line of inquiry — a family of 20 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 20
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- How do human annotators disagree systematically on ambiguous examples?
- Why do NLP benchmarks treat annotation disagreement as noise rather than signal?
- Can discourse communities collectively detect disruptions individual readers miss?
- Do high-disagreement items signal contested values or measurement noise?
- Why does consensus-seeking destroy information in normative but not factual tasks?
- What measurement artifacts emerge when annotators interpret the same question differently?
- Why are ground truth labels missing from unlabeled domain evaluations?
- What linguistic markers distinguish longer incorrect traces from correct ones?
- Can measuring semantic entropy help us detect unreliable generations?
- What information is lost when majority labels discard minority interpretations?
- Can meaning-level metrics like Semantic Entropy avoid length bias?
- Why do high-disagreement tasks benefit from broad rater pools over deep annotation?
- What replaces text-based expertise when surface markers become unreliable?
- What verification methods work for knowledge without stable referents?
- What happens to research goal-setting when a field lacks consensus on core terms?
- What makes a standardized artifact unit measurable across different research domains?
- What linguistic markers distinguish unfalsified corruption from other forms of error?
- How do local soundness signals work across different problem domains?
- Does adjudicated mean labels agree with each other or with some standard?
- What is craft-residue and why does its loss matter?