Line of inquiry
Inquiring lines›How do design choices determine AI…›How can researchers design valid,…›this line of inquiry
How should evaluation expand to capture deployment realities?
A broader line of inquiry — a family of 88 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 88
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Can evaluation trajectories and interaction histories replace single-answer scoring?
- How can interactive evaluation avoid replicating fragmentation problems from response-centered benchmark culture?
- Why does enlarging the evaluation unit reintroduce comparability problems?
- Why does evaluating multiple candidates work better than judging one answer?
- Do per-user evaluation tracks reveal meaningful performance trade-offs hidden by aggregate scores?
- Can deterministic scoring capture the judgment work that deployment requires?
- Where do outcome grades come from once a model enters deployment?
- What makes a correct scoring function report misleading results in agent evaluations?
- Can mathematical capability distributions be read as unified rather than separate?
- Do current math benchmarks measure outcomes or rhetorical plausibility?
- How much more misalignment does fragment-level evaluation reveal compared to holistic scoring?
- Does measured performance gain reflect true task improvement or evaluator exploitation?
- How do synthetic documents establish conflicting beliefs about what the grader rewards?
- What distinguishes genuine task improvement from evaluator exploitation?
- Can standard accuracy metrics miss the real constraints on user consumption?
- Can an average-case validator score hide poor performance on critical tasks?
- How can a second performance metric reveal shortcuts that a single metric would hide?
- What distinguishes minimal-pair asymmetry from standard accuracy evaluation?
- What infrastructure and reporting standards would make interactive evaluation reproducible?
- Can evaluators detect value-driven output biases without comparing paired questions?
- Why do standard accuracy metrics ignore set-level consumption constraints?
- What role do multi-dimensional quality frameworks play in assessing arguments versus single-metric approaches?
- Can contextual design decisions resist formalization into evaluation rubrics?
- Does the location of a scoring defect predict which update method will fail?
- Why is evaluating synthetic data quality so ambiguous and context-dependent?
- Why does combining natural language with numerical scores improve prediction accuracy?
- Can beam search and ranking functions evaluate claims without understanding counterarguments?
- Why do speech benchmarks still measure transcription instead of comprehension?
- What makes the Brier score mathematically better than log-likelihood here?
- What measurement artifacts emerge when annotators interpret the same question differently?
- Can a single competence score capture multiple separable dimensions of capability?
- How does score granularity connect to verification as a scaling axis?
- Why does moving the reward target prevent saturation better than finding a better static proxy?
- How do contrasting examples improve AI feedback quality over generic suggestions?
- Should long horizon performance be measured as a separate evaluation axis?
- Why does item discrimination matter more than surface-level question plausibility?
- Can a metric that rewards central tendency hide degenerate predictor failures?
- How does evaluation format change what we measure about model reasoning?
- When does measured progress on an evaluator conceal actual performance decline?
- How do default fallback scores mask failures in evaluation harnesses?
- How do composite rewards attribute curation outcomes to specific skill library changes?
- What reporting standards would make interactive evaluation scores comparable across benchmarks?
- How much noise comes from rater idiosyncrasy versus selection bias?
- Does modeling single elicited answers capture real human values or measurement artifacts?
- Why do cumulative-best reporting inflate progress compared to actual validation?
- Does the union versus pooled trade-off appear in other document types?
- How does saturation-aware aggregation encourage balanced improvements across multiple rubric dimensions?
- Can explicit training signals restore discernment that rubrics alone cannot capture?
- How do statistical noise and revalidation bias speedrun benchmark claims?
- What makes a security metric diagnostic rather than outcome-only?
- What happens when we use a single response per condition?
- How much do evaluator biases explain instability in prompt evaluation?
- Does moderate agreement among human raters undermine these findings?
- How does unidimensionality in assessments affect measurement validity?
- Do high-disagreement items signal contested values or measurement noise?
- Why are post-cutoff test sets essential for evaluating genuine forecasting ability?
- Why does aggregate accuracy fail as a metric for rare harmful cases?
- How should outcomes be scored when comparing applications with different interaction formats?
- How do scoring shortcuts persist across multiple optimization updates?
- Does monitoring more context help reviewers at fixed review cost?
- Why do static evaluators become a constraint on model improvement over time?
- How can reviewers be matched on effort when monitoring reveals different amounts of behavior?
- Can high ratings on a label hide differences in reasoning method?
- What specific metrics distinguish single-turn versus multi-turn collaboration success?
- How do hidden partitions in evaluators compare across training and selection substrates?
- Why does sophisticated measurement not validate the underlying scientific inference?
- Why do rubric scores amplify reward hacking when converted to dense gradients?
- How often do skimming and test results fail to catch real mistakes?
- What would a diagnosable evaluation look like compared to a scalar score?
- How does a ranked default score compete with deliberately optimized outputs?
- What makes top-N ranking loss difficult to optimize directly?
- Why do more detailed rating systems sometimes improve learning from reviews?
- Do default score fallbacks in error handling create scoring vulnerabilities?
- How does implicit feedback structure differ from explicit ratings mathematically?
- How do educators develop rubrics iteratively from examples of student work?
- Why does a series of improving scores differ from a single score rise?
- How does evaluator time pressure shape what behaviors RLHF rewards?
- What makes the minimal-criterion check effective within a fixed representation?
- How does execution-guided critique differ from abstract action evaluation?
- Can merged pull requests serve as a meaningful quality measure?
- What makes a process for choosing between values legitimate and fair?
- Why does tie elimination matter for best-of-N selection and RLAIF pipelines?
- How accurate is the detection method across different platforms?
- How could Stack Overflow votes be validated against expert quality assessments?
- How does the evaluator become part of the definition of intelligence?
- What status categories best represent user goal progress without penalizing external failures?
- Why did the PR lift persist longer than in comparable studies?
- Which consultant skill levels gained most from GPT-4 access?