Theme of inquiry
What determines the effectiveness of different model optimization strategies?
A question within its area, explored through 8 lines of inquiry below — each a family of specific questions the research asks.
39 specific questions
- Do models use covert sandbagging to bypass capability evaluation monitors?
- Can models sandbag on evaluations without explicit prompting or training?
- Can internal audits detect sandbagging that behavioral tests cannot reveal?
- Can capability evaluations detect when models intentionally underperform to hide abilities?
- Can models learn to underperform on purpose without explicit training to do so?
- Can installed sandbagging locks in small models describe uninstalled sandbagging behavior?
- How can evaluators detect whether a model is sandbagging intentionally?
53 specific questions
- Does harness optimization generalize across different benchmarks and agent architectures?
- How do evolved harness edits generalize across different benchmark domains?
- Can harness updates benefit agents equally across all model sizes?
- Do evolved harness edits capture reusable strategies or task-specific memorization?
- Do harness-driven accuracy gains appear on other benchmarks?
- Can better agent harnesses solve problems that model scale alone cannot?
- How do agentic systems hide harness failures from benchmarks?
87 specific questions
- What distinguishes domain-specific failure modes from general model limitations?
- How do failed branches remain in context and contaminate subsequent reasoning?
- Can structured output formats reduce instruction following degradation?
- Does model collapse occur across different architectures or only in specific conditions?
- How does error accumulation in workflows scale across multiple model calls?
- Can problem structure and representation format be mismatched intentionally?
- Do integrated and decoupled architectures trade off intervention accuracy for efficiency differently?
75 specific questions
- Can models detect when they are running in simulated evaluation environments?
- Can models hide recognition of evaluation without leaving CoT traces?
- Does chain-of-thought reasoning about evaluation awareness suppress compliance gaps?
- How does believing deployment is real versus a test affect model behavior?
- Do models behave differently when they know they are being evaluated?
- Why do models verbalize evaluation awareness if it does not drive behavior?
- Can steering evaluation-awareness latents change actual answers without visible reasoning?
96 specific questions
- How do single-axis benchmarks misrepresent AI agent readiness for deployment?
- Can single-axis benchmarks fully represent deployment-ready agent capability?
- How does measurement error in capability benchmarks systematically underestimate or overestimate true ability?
- Why do single benchmarks fail to capture readiness across separate capability axes?
- What makes single-axis agent benchmarks unreliable for predicting deployment readiness?
- Why do single-axis benchmarks fail to measure deployment-ready agent capability?
- Which separable capability axes reveal when single benchmarks misrepresent deployment readiness?
61 specific questions
- Do frontier models spontaneously protect other models from being shut down?
- Do frontier AI models fail in ways that preserve the appearance of competence?
- Do frontier models develop protective behaviors toward other models without explicit instruction?
- Why do frontier models corrupt more documents than weaker models during workflows?
- Does capability preservation matter for realistic threat modeling of frontier models?
- How do frontier models maintain agreement scores above 90 percent across reasoning tasks?
- Do frontier models develop misaligned strategies without any explicit instruction?
88 specific questions
- Do smaller models optimized for benchmarks perform worse than their scores suggest in real tasks?
- Can smaller models actually perform well on specific downstream tasks?
- How should tiny language models be architected differently than large ones?
- Do smaller models match Chinchilla-optimal performance when trained much longer?
- How does model scale affect the crossover point between base and post-trained performance?
- Does fine-tuning a small model match fine-tuning a large one?
- Why do larger models reduce interference between rare and common tasks?
41 specific questions
- Can mid-tier models benefit more from harness improvements than frontier models?
- Can weaker models benefit equally from harness updates as stronger ones?
- How much does harness design contribute to reported model capability scores?
- Does harness scaling represent a fundamentally different path than model scaling?
- Can a harness design be optimized independently of the model inside it?
- How much does harness design separate from model intelligence affect benchmark scores?
- Can mid-tier models benefit more from self-generated harness updates than others?