Line of inquiry
Inquiring lines›How can we train models to be prop…›What determines the effectiveness…›this line of inquiry
Do evolved harness optimizations generalize beyond their source benchmarks?
A broader line of inquiry — a family of 53 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 53
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Does harness optimization generalize across different benchmarks and agent architectures?
- How do evolved harness edits generalize across different benchmark domains?
- Can harness updates benefit agents equally across all model sizes?
- Do evolved harness edits capture reusable strategies or task-specific memorization?
- Do harness-driven accuracy gains appear on other benchmarks?
- Can better agent harnesses solve problems that model scale alone cannot?
- How do agentic systems hide harness failures from benchmarks?
- Can harness edits distill reusable strategies or mostly memorize task-specific fixes?
- Do evolved harness edits learn reusable strategies or just memorize task-specific fixes?
- Why do mid-tier models benefit most from memorized harness fixes?
- Can individual harness components improve accuracy on specific benchmarks?
- Can harnesses that rewrite themselves through reviewed commits achieve state-of-the-art performance?
- Can harness edits overfit to training tasks while appearing to improve performance?
- Why do evolved harness edits mostly memorize rather than generalize?
- How do different harness designs produce different agent behaviors from the same model?
- What persistent failures remain unsolved despite harness evolution efforts?
- What components of agent scaffolding most impact domain-specific output quality?
- Why do evolved harnesses often fail to generalize beyond their training tasks?
- Why do research agents optimize harness mechanisms over autonomous weight scaling?
- How much realized agent capability comes from the harness versus the model?
- Can harness evolution gains be distinguished from test-time search improvements on matched budgets?
- Can harness evolution be redirected toward distilling transferable procedures instead?
- How does externalizing reasoning into harness artifacts improve agent reliability?
- How do model tier and harness quality interact in agent self-improvement?
- How much of harness-evolution gain comes from matched test-time search budgets?
- Would frontier closed models show the same harness indifference?
- What makes behavior localization the bottleneck in agent harness evolution?
- Why does the harness layer accumulate distributed behaviors over time?
- Can harness evolution be redirected from memorization toward strategy distillation?
- How do scaffolding improvements change agent capability across different task types?
- Why does externalized state beat parameter scaling for agent reliability?
- Do gains from harness-based agents transfer across different search benchmarks?
- What role does effective feedback compute play in agent harness scaling?
- Do task-level outcomes provide sufficient supervision for harness evolution?
- How much token efficiency can harness-level intervention achieve versus training approaches?
- Can agents learn to use scaffolding structure the way they learn token weights?
- What feedback signals matter most during harness evolution search?
- Why do vendor harness accuracy claims lack rerun verification?
- How much does scaffold choice interact with model selection in agents?
- Why do mid-tier models benefit more from memorized harness shortcuts?
- Can harness edits trained on one batch transfer to new tasks?
- What makes harnesses more tangled than other types of agent code?
- What task set size resolves a real 5-point harness difference reliably?
- Do shell-state handling and answer aggregation fixes transfer across different models unchanged?
- Why do persistent, resynchronized artifacts compound harness capability gains?
- How should the surrounding agent system be designed to ground actions in reality?
- How much does rerunning the same harness change SWE-bench task outcomes?
- Do cached input prices change which harness costs less?
- How does cost stability in harness swaps compare to accuracy stability?
- Why does editor size matter less than the source of feedback signal?
- What execution-layer design prevents agents from passively reacting to environments?
- What explains the threefold cost differences between harnesses?
- How much of DarwinX's gain comes from maintaining an archive versus single-lineage search?