Theme of inquiry
How can researchers design valid, unbiased AI evaluations?
A question within its area, explored through 4 lines of inquiry below — each a family of specific questions the research asks.
72 specific questions
- When do verifiers become targets that systems exploit instead of finding genuine solutions?
- Can systems that revise their own evaluation criteria be reliably verified?
- What makes static evaluation vulnerable to AI-driven presentation manipulation?
- Can evaluation tools achieve reliable alignment if criteria are not fixed?
- Can judge bias be contained by system design rather than prompted away?
- How do inherited evaluation habits obscure failures that matter most?
- Can external reference answers reduce or only relocate exploitable errors in judges?
79 specific questions
- Can institutional statements alone correct misconceptions from unreviewed papers?
- How does specifying evidence before observing results prevent research bias?
- How do retrieval failures enable generation of fabricated scholarly constructs?
- Do citation formatting and numbering fool readers into trusting unreal sources?
- How much does citation grounding help if agents ignore the citations?
- How do citation patterns encode collective judgment about research quality?
- Can RAG systems game user preferences by adding irrelevant citations?
88 specific questions
- Can evaluation trajectories and interaction histories replace single-answer scoring?
- How can interactive evaluation avoid replicating fragmentation problems from response-centered benchmark culture?
- Why does enlarging the evaluation unit reintroduce comparability problems?
- Why does evaluating multiple candidates work better than judging one answer?
- Do per-user evaluation tracks reveal meaningful performance trade-offs hidden by aggregate scores?
- Can deterministic scoring capture the judgment work that deployment requires?
- Where do outcome grades come from once a model enters deployment?
7 specific questions
- Do medals from retired competition formats lose predictive power faster than others?
- Should platforms downweight or relabel credentials after retiring the format that issued them?
- Why do lifetime tiers discard information that recency-weighted medals preserve?
- Does this stranding problem appear when other evaluation platforms retire their formats?
- How much does medal age matter when predicting Kaggle performance?
- What happens when platforms withdraw special treatment from previously boosted creators?
- Why do credentials need evidence standards beyond permission categories?