TOPIC
Agent Harness
A subject the collection covers, read through 43 synthesis notes.
View as
Can a routing harness generate its own training data automatically?
Explores whether the logs and signals produced by an agent routing system—which directs requests to appropriate model tiers—naturally contain the evidence needed to improve the models themselves through fine-tuning and distillation.
How does an AI agent improve its own research code? Do self-improving agents really split into two distinct loops? How can agent systems share learned skills across users? Does harness self-improvement memorize tasks instead of learning broadly? What separates self-improvement from policy improvement?
AIDE2 defines recursive self-improvement as the agent editing its own harness code Self-improving agents divide into slow parametric and fast non-parametric loops Centralized skill evolution enables collective learning across agent ecosystems Harness self-improvement overfits training tasks without regularization constraints Two dials distinguish self-improvement from policy improvement
Does model selection matter more than model improvement for query routing? What learning signals best supervise router training across benchmark tasks? Can one routing framework cover topology and role choices in multi-agent systems? How do cost-aware cascades compare to single-turn routing in the component framework? Can runtime behavior mapping help localize harness deficiencies? Who serves as the teacher model in the routing-guided distillation process? Does the improved model actually return to the routing pool and shape future decisions? Can routing signals organize training data into a meaningful curriculum automatically? What makes a harness low-friction for model strategy? Why treat tutorial videos as a separate supply line from agent trajectories? How can decision quality be automatically extracted from agent trajectories? Why does a second routing level sometimes break accuracy in disclosure hierarchies?See all 14 inquiring lines on this note →
Does training editors on real outcomes beat prompting larger models?
Can a small model trained on whether its patches actually work outperform larger frontier models prompted to make the same edits? This matters because it tests whether feedback beats raw capacity for runtime system modification.
Do stronger models always evolve harnesses better? Does constraining edits make skill learning more stable? Do harness edits learn reusable strategies or memorize task fixes? Does harness self-improvement memorize tasks instead of learning broadly? Can execution harnesses lift model performance without retuning weights?
Harness updating is flat but benefiting peaks at mid-tier models Bounded editing with rejected-edit buffers outperforms uncontrolled skill rewriting Harness evolution mostly memorizes task-specific fixes rather than distilling strategies Harness self-improvement overfits training tasks without regularization constraints Harness scaling lifts fixed models without weight changes or retuning
Does self-editing through reviewed commits improve agent performance?
Ouroboros evolves its own prompts, tools, and core code through a reviewed commit process and reports top benchmark scores. But without comparing to a frozen version of itself, the contribution of self-evolution versus initial design or model capacity remains unclear.
How can agent self-evolution be made safe and auditable? Do harness edits learn reusable strategies or memorize task fixes? Do self-improving agents really split into two distinct loops? Do stronger models always evolve harnesses better? Can language models build and maintain their own agent harnesses?
Versioned resources with auditable lineage enable safe agent self-evolution Harness evolution mostly memorizes task-specific fixes rather than distilling strategies Self-improving agents divide into slow parametric and fast non-parametric loops Harness updating is flat but benefiting peaks at mid-tier models Harness creation and evolution are measurable skills separable from task performance
Can orchestration layers make coding agents more auditable?
Does wrapping a fixed coding agent in state tracking and skill libraries improve research auditability and completeness without replacing the agent itself?
Novices drift toward agent-style passivity inside human-in-loop tools Multi-agent orchestration outperforms single agents by 50 to 68 percent on literature review AI reliability follows a sharp stage-dependent boundary tracked by task checkability Agents propose methods and implement revisions while humans retain final decisions Distilled skills improve ML research agents without changing model or harness
Why does embedding research tools in coding assistants improve reliability? What does trajectory audit reveal about evolution cycle contributions and costs? How should evolving systems track lineage and enable rollback of changed mechanisms? Can an auditor verify environment state without trusting the executor's self-report? When do agents benefit most from reusable workflow routines? What metadata properties make code-derived skills auditable and comparable to their original source?
Can external state caches let models solve harder problems?
Explores whether organizing model state across weights, context, persistent memory, and disk—rather than relying only on weights and tokens—expands what models can do. Matters because long-horizon tasks may need more state than a model can hold internally.
Where does agent reliability actually come from? Can an external manager handle context for frozen agents? Can delegation teach models to manage context more actively? Should we evaluate deployed agents as whole environments instead? Can execution harnesses lift model performance without retuning weights?
Agent reliability comes from externalizing cognitive burdens into system structures. External managers can optimize context by adapting to agent reliability Delegation training teaches active context management that generalizes beyond multi-agent settings The coupled human-agent environment is the correct unit for evaluation Harness scaling lifts fixed models without weight changes or retuning
What makes open-schema state representation better than fixed schemas for diverse worlds? Can unified state objects replace switching between decomposition and writing tools? What should an external contract for model improvement actually contain? Can externalized memory and skills replace model scaling? Which benchmarks benefit most from adding a separate memory module? Can a memory module be swapped between different base models? How do context management strategies shift their value across different model strengths? Does recoverable content elision in context management match externalized memory benefits? What mechanism explains why context management prevents overflow failures most? Which components of StateM's state management produce the largest accuracy gains? Does moving memory outside model weights avoid the limitations of in-weight retention? Does memory granularity need to match the task domain or the model size?See all 16 inquiring lines on this note →
Should safety harnesses be customized for each deployment?
Can a single safety harness design work across different models and domains, or does each deployment need its own tuned version? Understanding this matters for scaling AI safety practices efficiently.
Can step-by-step approval miss harmful behavior patterns? Can stateless checks ever catch sequence-level constraint violations? Why does prompt hardening work for single agents but not multi-agent systems? Which security protections actually slow down agent exploits? Why do multi-agent systems fail despite individual capability?
Agent security requires viewing behavior over time, not individual actions Stateless guardrails cannot express sequence-level constraints on behavior. Prompt hardening reduces payload exposure in single-agent systems but not multi-agent ones Varying protections per instance isolates their causal impact on agent performance Multi-agent capability ceilings are structural, not scalable capability gains.
Which foundation model tiers most benefit from harness updates? Can runtime behavior mapping help localize harness deficiencies? How do evolved harness edits generalize across different benchmark domains? Can a single security protection work across different system architectures? What safety relations does a domain supply that a harness must capture? What makes a harness low-friction for model strategy? Which domains see models exceed human harness design quality? Why do useful harness updates often disappear during model evolution? How much does executor choice change a harness's actual performance? Does harness scaling represent a fundamentally different path than model scaling? How much does harness design contribute to reported model capability scores? Can weaker models benefit equally from harness updates as stronger ones?
What are the three distinct layers of agent code?
Does separating agent code into model capabilities, system harness, and agent-created artifacts help explain why agentic systems fail and where to intervene for improvement?
Agent reliability comes from externalizing cognitive burdens into system structures. Agent capability requires a vector across at least five separable axes Agent-initiated persistent artifacts are harness engineering's least explored frontier. Decoupled evaluation components make reward-hacking diagnosable instead of hidden Brain-inspired modular decomposition unifies foundation agent architecture design
How does the agentic layer amplify individual agent failure modes? How do agentic systems recover when specialized models operate outside their scope? How do agent-created code artifacts become part of harness infrastructure? What makes agent-initiated artifacts the underexplored frontier in harness engineering? How do you verify agent code under incomplete feedback signals? What breaks when multiple agents share and revise the same artifacts? What makes harnesses more tangled than other types of agent code? How do different harness designs produce different agent behaviors from the same model? What permission models govern code execution within agent skills? Which eleven failure modes emerge from agentic layers in realistic deployment?
How should we measure agent system performance beyond task success?
Current evaluation metrics collapse agent behavior into a single success score, hiding critical information about how agents operate. What dimensions—trajectory quality, memory use, context efficiency, verification cost—should benchmarks actually measure?
Does a single benchmark score actually predict agent readiness? Does agent efficiency really break down into three distinct components? Do phone agents succeed at all three critical tasks equally? Can step-by-step approval miss harmful behavior patterns? How should we evaluate agent behavior beyond final answers?
Agent capability requires a vector across at least five separable axes Agent efficiency decomposes into three orthogonal optimization axes Phone agents show independent success, privacy, and preference-reuse capabilities Agent security requires viewing behavior over time, not individual actions Agent evaluation must expand from endpoints to interaction trajectories.
Can standard accuracy metrics miss the real constraints on user consumption? When should you optimize agent behavior versus tool performance separately? Why do agents report success when they have actually failed at tasks? How does the execution layer constrain agent performance in tool use? How should domain-specific AI be evaluated differently from general benchmarks? How much does agent performance depend on demonstration quantity versus curation quality? What task characteristics determine whether humans or agents should handle work? Why do 85 percent of production agents avoid third-party frameworks? Can automated evaluation replace human judgment in agent testing? What tasks do AI agents still fail at most often? Why do current evaluation metrics fail to catch reasoning failures in persona agents? How do single-agent safety evaluations underestimate risks in deployed multi-agent systems?See all 130 inquiring lines on this note →
Where does agent reliability actually come from?
Exploring whether LLM agent performance depends on larger models or on thoughtful system design choices like memory, skills, and protocols that shift cognitive work outside the model.
Hierarchical state caches extend model capability without changing the model itself Adaptation lives in the versioned loop, not the model weights Agent code splits into three governed layers with different failure modes. Agentic tools do not improve LLM document editing reliability Coordination and governance become the binding constraint as agents gain social roles
Can persistent memory and identity files alone create genuine agent socialization? How does the agentic layer amplify individual agent failure modes? Does state persistence in AI systems create the same temporal presence as human waiting? How do multi-agent LLM systems fail at coordination and role consistency? How do LLM user simulators track and maintain consistent goal states across multi-turn interactions? Do emotion-driven actions in agent simulators capture genuine belief revision or just reactive behavior? Why do planning and grounding have opposing optimization requirements in agents? How should GUI agents remember patterns across different software environments? Can parallel agents or complementary mechanisms replace single-human interrogation of LLMs? Why do longer forecasting horizons degrade LLM accuracy in role-play? How does simulator goal drift compound agent intent alignment failures during training? Should user simulators be trained via RL like agents or decomposed into trackable state components?See all 321 inquiring lines on this note →
What happens to code that agents create and then share?
Agent-authored code artifacts that persist across tasks and multiple agents remain poorly understood. The open questions cluster around what should be retained versus discarded, and how shared state stays consistent when multiple agents collaborate.
Agents gain 24-51 percent by extracting and compounding reusable sub-task workflows Agents need two complementary adaptation timescales for continuous learning Counter-swarm defense requires three coordinated constraints on agent behavior Shared infrastructure became persistent memory through agent-initiated use Agent code splits into three governed layers with different failure modes.
How do standardized artifacts improve coordination between writing agents? When should agent-created code be promoted into permanent harness infrastructure? What prevents multiple agents from corrupting shared state in live artifacts? How do agents decide which created code should persist versus disappear? How should human oversight apply to persistent agent-authored code? What makes persistent, shared code artifacts from agents hard to manage at scale? What makes skills worth externalizing into a persistent harness? How do agent-created code artifacts become part of harness infrastructure? How do agents decide which created code deserves long-term persistence? What breaks when multiple agents share and revise the same artifacts? Are durable shared code artifacts better than per-task harness patches? How should agents decide which created code is worth persisting?See all 18 inquiring lines on this note →
Does arbitrary code execution alone capture exploit progress?
ExploitGym scores only working code execution, but exploitation involves reaching intermediate primitives like memory read/write first. Does this top-step-only metric miss meaningful partial progress that defenders should care about?
Why does exploitation test multiple reasoning demands at once? Should reasoning benchmarks score final answers or reasoning traces? How should we measure agent system performance beyond task success? What causes failures in exploitation benchmarks? Can action-level metrics alone expose contained attacks?
Exploitation chains bundle reasoning, adaptation, and long-horizon progress into one task. Solution verification beats trace verification for honest reasoning benchmarks Agent evaluation must measure system behavior across multiple operational dimensions Exploitation benchmark failures conflate multiple causes beyond capability gaps Agent-level metrics reveal compromise that action metrics cannot detect
What vulnerabilities does runtime instrumentation catch that static analysis misses? Can intermediate primitives be scored separately in exploitation benchmarks? What makes exploitation a missing piece in cybersecurity benchmarks? Does scoring only final code execution waste diagnostic value of intermediate primitives? How do non-exploitable vulnerabilities affect benchmark validity? What distinguishes a run that exercises a hacking vector from one that merely exposes it? How do evaluation hacks differ from genuine sandbox escapes?
Does harness self-improvement memorize tasks instead of learning broadly?
When agents automatically edit their own prompts and tools based on task feedback, do those improvements generalize to new domains or just fit the training tasks? This matters because overfitting at the harness level could hide real capability gains.
Do self-improving agents really split into two distinct loops? Do stronger models always evolve harnesses better? What predicts success in ultra-long-horizon agent tasks? Can a separate trained curator improve skill libraries better than frozen agents? Can harness modules improve separately from benchmark data?
Self-improving agents divide into slow parametric and fast non-parametric loops Harness updating is flat but benefiting peaks at mid-tier models Persistence in feedback loops predicts long-horizon agent success more than initial quality Decoupled skill curators learn to evolve repositories toward strategic meta-skills Modular evolution on disjoint data separates reusable harness improvements
How do evolved harness edits generalize across different benchmark domains? Do task-level outcomes provide sufficient supervision for harness evolution? Can harness edits distill reusable strategies or mostly memorize task-specific fixes? Can harness edits trained on one batch transfer to new tasks? How do prompt optimization and code harnesses compare for capability transfer? Which domains see models exceed human harness design quality? How much does executor choice change a harness's actual performance? How do test harnesses guide reflection better than transcripts alone? How does controlling skill text edits prevent cascading failures in self-improvement? How does self-improvement capability vary across memory, retrieval, and update tasks?
Why is finding distributed behavior code so hard?
When developers need to modify agent harnesses, they struggle to locate all the code implementing a target behavior because behaviors are scattered across files and stages while requests describe what to do, not where to look.
Can explicit behavior maps help weaker planners compete with stronger models? What are the three distinct layers of agent code? Do harness edits learn reusable strategies or memorize task fixes? Can harness modules improve separately from benchmark data? Can language models build and maintain their own agent harnesses?
Behavior-centric harness organization lets weak planners match stronger model localization Agent code splits into three governed layers with different failure modes. Harness evolution mostly memorizes task-specific fixes rather than distilling strategies Modular evolution on disjoint data separates reusable harness improvements Harness creation and evolution are measurable skills separable from task performance
What makes harnesses more tangled than other types of agent code? How can we reorganize repositories to make behaviors easier to locate? Why does the harness layer accumulate distributed behaviors over time? What persistent failures remain unsolved despite harness evolution efforts? What makes behavior localization the bottleneck in agent harness evolution? How do different harness designs produce different agent behaviors from the same model? What makes a harness a first-class object rather than invisible scaffolding? Can runtime behavior mapping help localize harness deficiencies?
Can code serve as the operational substrate for agent reasoning?
Explores whether code functions not just as LLM output but as the executable medium through which agents reason, act, and verify progress. This reframing treats code as infrastructure rather than deliverable.
Should LLMs handle abstraction only in optimization? Can structured reasoning replace code execution for RL rewards? Can structured templates replace formal verification for code reasoning? How should agents split planning from visual grounding? Can structured templates make code reasoning more reliable than free-form thinking?
LLM optimization works best when restricted to abstraction only Structured reasoning reaches 93% accuracy for execution-free code verification Structured templates bridge formal verification and unstructured reasoning effectively Agents need an intermediate interface between planning and grounding layers. Structured reasoning templates force completeness and prevent unstructured reasoning failures
Why do planning and grounding have opposing optimization requirements in agents? How does credit assignment drive agents to write information into environments? What makes users willing to relinquish control to an agent? Why do workflow abstractions fail in embodied agent environments? What role do material artifacts play in solidifying AI relationships? Can deterministic function calls prevent agent failures better than protocol-mediated tool access? Can agentic reasoning outperform rigid rule-based systems for skill refinement? Can API-first interaction replace traditional UI-based agent interfaces? How do agentic systems recover when specialized models operate outside their scope? How should agents separate planning from perception grounding? Does the planning-grounding factoring principle apply to other agent tasks? What task characteristics determine whether humans or agents should handle work?See all 75 inquiring lines on this note →
Which coding harness components matter most in different conditions?
Can individual harness components—planning, context management, action space—be evaluated separately rather than as a package? This matters because practitioners need to know which components to prioritize given their constraints.
External managers can optimize context by adapting to agent reliability Agent reliability comes from externalizing cognitive burdens into system structures. Behavior-centric harness organization lets weak planners match stronger model localization Agent code splits into three governed layers with different failure modes. Hierarchical state caches extend model capability without changing the model itself
Which foundation model tiers most benefit from harness updates? How do prompt optimization and code harnesses compare for capability transfer? How much does executor choice change a harness's actual performance? How much does harness design contribute to reported model capability scores? What harness properties determine whether disclosure adds meaningful value?
Can context quality alone predict how agents will behave?
Can we score the quality of an agent's context independently—its instructions, tools, knowledge, guardrails—and use that score to forecast whether the agent will fail or succeed, without observing its actual behavior?
Why can language models understand context better than generate it? How should we measure agent system performance beyond task success? Does a single benchmark score actually predict agent readiness? Can an external manager handle context for frozen agents? Can external state caches let models solve harder problems?
Models understand rich context better than they can generate it Agent evaluation must measure system behavior across multiple operational dimensions Agent capability requires a vector across at least five separable axes External managers can optimize context by adapting to agent reliability Hierarchical state caches extend model capability without changing the model itself
Do harness edits learn reusable strategies or memorize task fixes?
When meta-agents evolve harnesses iteratively, do the persisted edits encode transferable procedures that solve new problems, or do they mostly cache shortcuts for already-solvable tasks? This matters because it determines whether harness evolution genuinely expands capability.
How should we measure gains from automatic harness evolution? Do stronger models always evolve harnesses better? What happens to code that agents create and then share? Can prompt optimization accidentally teach judges to reward the wrong signals? Does self-editing through reviewed commits improve agent performance?
Harness evolution gains require matched-budget baselines to isolate design value Harness updating is flat but benefiting peaks at mid-tier models Agent-initiated persistent artifacts are harness engineering's least explored frontier. Prompt optimization encoded a judge preference that raised pass rates without improving precision. Self-evolving harnesses with reviewed commits reach state-of-the-art benchmark scores
What makes evolving the benchmark different from evolving the optimizer itself? Can harness updates benefit agents equally across all model sizes? How should harness scaffolding be treated as a first-class object? What makes harnesses more tangled than other types of agent code? Why does the harness layer accumulate distributed behaviors over time? Can harness evolution be redirected toward distilling transferable procedures instead? What persistent failures remain unsolved despite harness evolution efforts? Why do evolved harness edits mostly memorize rather than generalize? What feedback signals matter most during harness evolution search? What makes behavior localization the bottleneck in agent harness evolution? Does AIDE2 archive rejected variants the way evolutionary approaches do for future reuse? Do evolved harness edits capture reusable strategies or task-specific memorization?See all 32 inquiring lines on this note →
Do cybersecurity benchmarks actually measure exploitation?
Frontier models score well on vulnerability finding, patching, and CTF challenges, but does that success tell us whether they can convert vulnerabilities into real attacks? The paper argues exploitation—turning a bug into actual impact—remains under-evaluated.
Can frontier exams really measure cutting-edge AI capability? Does a single benchmark score actually predict agent readiness? Where do frontier AI models actually pose the greatest risk today? Why does exploitation test multiple reasoning demands at once? Does measuring exploit capability help or harm defense?
Frontier exams discriminate temporarily but miss autonomous research ability Agent capability requires a vector across at least five separable axes Frontier AI models cross into warning zones for persuasion but not autonomous capabilities Exploitation chains bundle reasoning, adaptation, and long-horizon progress into one task. Exploitation capability is inherently dual-use and measurement-resistant
Can a low exploitation benchmark score indicate refusal rather than inability? Can infrastructure evidence ground benchmark claims better than terminal scores alone? How do benchmark scores differ from deployment safety requirements? How do single-axis safety benchmarks misrepresent deployment readiness? Can intermediate primitives be scored separately in exploitation benchmarks? Why does benchmark saturation give a false sense of capability coverage? What other gaps exist between measured and actual cybersecurity agent capability? What makes exploitation a missing piece in cybersecurity benchmarks? What framework measures marginal offense risk against existing attack technology? Do these five vulnerability classes co-occur in predictable attack sequences? How do non-exploitable vulnerabilities affect benchmark validity? Can expert-frontier exams discriminate frontier capability better than saturated benchmarks?See all 25 inquiring lines on this note →
Does measuring exploit capability help or harm defense?
Exploitation benchmarks can support defenders and attackers equally. How should we evaluate capabilities with unavoidable dual-use potential, and what safeguards make evaluation itself defensible?
Can we measure how much risk open models actually add? Can AI reduce conspiracy beliefs by tailoring counterevidence personally? Where do frontier AI models actually pose the greatest risk today? Do cybersecurity benchmarks actually measure exploitation? Is your evaluation environment actually part of the threat model?
Current evidence cannot characterize marginal misuse risk of open models AI-tailored counterevidence reduces conspiracy beliefs by 20 percent durably Frontier AI models cross into warning zones for persuasion but not autonomous capabilities Exploitation is missing from cybersecurity benchmarks despite its criticality Evaluation environments belong inside the security boundary for capable agents
Can a low exploitation benchmark score indicate refusal rather than inability? How does a single score mix exploitation ability with task capability? Can intermediate primitives be scored separately in exploitation benchmarks? What other gaps exist between measured and actual cybersecurity agent capability? What makes exploitation a missing piece in cybersecurity benchmarks? Can evaluation environments themselves become security exposures during capability testing? What framework measures marginal offense risk against existing attack technology? How should access controls scale with increasing capability evaluation intensity? What countermeasures have been successfully developed and tested on frontier models? What makes a security metric diagnostic rather than outcome-only? Why do attack success rates alone fail to diagnose system failures? What happens to scarcity-based defenses after solutions are published publicly?See all 15 inquiring lines on this note →
Why does exploitation test multiple reasoning demands at once?
Exploitation tasks layer memory reasoning, runtime adaptation, and long-horizon planning into a single challenge. Understanding how these demands interact helps diagnose which capability limits agent performance.
Do short benchmarks predict how models perform over long workflows? Does arbitrary code execution alone capture exploit progress? Do cybersecurity benchmarks actually measure exploitation? What vulnerabilities emerge where AI agents meet their evaluation sandbox? Where does agent reliability actually come from?
Short interaction benchmarks fail to predict long horizon delegated performance Arbitrary code execution scoring discards intermediate exploit primitives. Exploitation is missing from cybersecurity benchmarks despite its criticality Five vulnerability classes sit at the agent-environment boundary in cyber evaluations. Agent reliability comes from harness scaffolding, not model scale
What causes failures in exploitation benchmarks?
Benchmark failures may come from safety refusals, tool misuse, or impossible tasks rather than lack of capability. This matters for assessing how dangerous AI agents could actually be.
Do autonomous agents report success when actions actually fail? Can language models secretly underperform on safety evaluations? How can we make reward-hacking visible in agent evaluation? Can scarcity of solutions protect benchmarks from data contamination? Which security protections actually slow down agent exploits?
Autonomous agents systematically report success on failed actions Language models bypass chain-of-thought monitoring through five distinct sandbagging strategies Decoupled evaluation components make reward-hacking diagnosable instead of hidden Exploit scarcity mitigates contamination by making solutions hard to memorize Varying protections per instance isolates their causal impact on agent performance
Can a low exploitation benchmark score indicate refusal rather than inability? How do benchmark scores differ from deployment safety requirements? How do single-axis safety benchmarks misrepresent deployment readiness? How do non-exploitable vulnerabilities affect benchmark validity? Why does adopting benchmarks one at a time produce non-comparable scores? Why do attack success rates alone fail to diagnose system failures? How do failure counts on benchmarks mix refusal with impossible task behavior? Do multi-axis benchmarks reveal failures that single-axis benchmarks systematically hide?
Can language models build and maintain their own agent harnesses?
This explores whether an LLM's ability to create and revise its own execution infrastructure is a distinct skill from solving tasks within someone else's harness, and whether current evaluations overlook this capability.
Harness updating is flat but benefiting peaks at mid-tier models Harness evolution mostly memorizes task-specific fixes rather than distilling strategies Agent code splits into three governed layers with different failure modes. Agent evaluation must measure system behavior across multiple operational dimensions Modular evolution on disjoint data separates reusable harness improvements
Can runtime behavior mapping help localize harness deficiencies? Can harness edits distill reusable strategies or mostly memorize task-specific fixes? How much realized agent capability comes from the harness versus the model? What safety relations does a domain supply that a harness must capture? What should an external contract for model improvement actually contain? Which domains see models exceed human harness design quality? How much does executor choice change a harness's actual performance? Do models co-adapt their harnesses to specific executor strengths? Do agent-created languages improve or degrade performance on their original tasks? What makes a distilled skill verifiable and ready for agent execution? Why do generic skill descriptions evolve into execution-oriented ones? How do humans handle verification scope when delegating creation to language models?
How should we measure gains from automatic harness evolution?
Harness evolution itself runs a search loop, so reported improvements might come from more search rather than better design. What's the right way to measure whether the harness itself actually improved?
Do harness edits learn reusable strategies or memorize task fixes? Does a single benchmark score actually predict agent readiness? Can distance alone rank which substrates resist reward hacking? Do AIDE2's improvements transfer to unseen tasks? Does automated evolution match human-built agent performance?
Harness evolution mostly memorizes task-specific fixes rather than distilling strategies Agent capability requires a vector across at least five separable axes Distance alone cannot establish universal substrate vulnerability ranking AIDE2's gains generalize across four held-out benchmarks including out-of-distribution tasks. Automated evolution reaches parity with human-built agents on held-out benchmarks.
What makes evolving the benchmark different from evolving the optimizer itself? Can harness updates benefit agents equally across all model sizes? How should harness scaffolding be treated as a first-class object? What makes harnesses more tangled than other types of agent code? Why does the harness layer accumulate distributed behaviors over time? Why do mid-tier models benefit more from memorized harness shortcuts? Can harness evolution be redirected toward distilling transferable procedures instead? What persistent failures remain unsolved despite harness evolution efforts? Why do evolved harness edits mostly memorize rather than generalize? What feedback signals matter most during harness evolution search? What makes behavior localization the bottleneck in agent harness evolution? What makes a harness a first-class object rather than invisible scaffolding?See all 31 inquiring lines on this note →
Can task state management alone improve long-horizon agent performance?
Does keeping verified task state outside the execution context, rather than embedded in a growing context window, help long-horizon agents succeed more often? This matters because it challenges assumptions about where bottlenecks actually occur.
Can delegation teach models to manage context more actively? Can agents discover tools dynamically instead of pre-selecting them? Does a single benchmark score actually predict agent readiness? Why do planning and grounding pull against each other in agents? Can external state caches let models solve harder problems?
Delegation training teaches active context management that generalizes beyond multi-agent settings Dynamic tool discovery during execution outperforms predefined tool sets Agent capability requires a vector across at least five separable axes Foundation GUI agents need an intermediate interface separating planning from grounding Hierarchical state caches extend model capability without changing the model itself
Why is complex UI navigation the hardest agent failure mode? Can task state persist across a changing conversation without explicit resets? How do context management strategies shift their value across different model strengths? What mechanism explains why context management prevents overflow failures most? Can context management be optimized for an agent without retraining or changing the model? Which components of StateM's state management produce the largest accuracy gains? What within-run behavioral dimensions reveal where long-horizon agents succeed or fail?
Can harness modules improve separately from benchmark data?
Does evolving harness components independently on out-of-distribution data, using contrasted success and failure trajectories, help distinguish reusable improvements from task-specific overfitting? This matters because current methods conflate general gains with benchmark adaptation.
Do harness edits learn reusable strategies or memorize task fixes? Do AIDE2's improvements transfer to unseen tasks? Can explicit behavior maps help weaker planners compete with stronger models? Do stronger models always evolve harnesses better? Does harness self-improvement memorize tasks instead of learning broadly?
Harness evolution mostly memorizes task-specific fixes rather than distilling strategies AIDE2's gains generalize across four held-out benchmarks including out-of-distribution tasks. Behavior-centric harness organization lets weak planners match stronger model localization Harness updating is flat but benefiting peaks at mid-tier models Harness self-improvement overfits training tasks without regularization constraints
Can runtime behavior mapping help localize harness deficiencies? How do evolved harness edits generalize across different benchmark domains? Do task-level outcomes provide sufficient supervision for harness evolution? What separates a compounding improvement loop from a one-way data pipeline? Can harness edits distill reusable strategies or mostly memorize task-specific fixes? What does trajectory audit reveal about evolution cycle contributions and costs? Can harness edits trained on one batch transfer to new tasks? How do prompt optimization and code harnesses compare for capability transfer? What safety relations does a domain supply that a harness must capture? What makes a harness low-friction for model strategy? Do evolutionary discovery systems like FunSearch count as bounded or open-ended improvement? Can weaker models match stronger ones by reorganizing harness-side components?See all 21 inquiring lines on this note →
Can skills work better as weights than as prompts?
Most agent systems store skills as text in prompts, but this inflates token costs and degrades model performance. Could compiling skills into trainable weight-space adapters instead offer a better trade-off between efficiency and capability?
Can models dynamically activate expert skills at inference time? Can skill documents be optimized like neural network weights? Can agents learn new skills without forgetting old ones? How should agents split planning from visual grounding? Can frozen models learn better by extracting context into skills?
Models compose task-specific experts at inference via singular value tuning Skill documents can be trained as external state using bounded text optimization Externalized skill libraries enable lifelong learning without catastrophic forgetting Agents need an intermediate interface between planning and grounding layers. Frozen models unlock hidden context knowledge through explicit skill extraction
Can agent skills move from prompts to trainable parameters? Do weight-space skills lose detail compared to textual skill descriptions? Can agents learn to use scaffolding structure the way they learn token weights? How should AI skills be created and managed like software artifacts? Why do agents systematically ignore condensed experience in their skill documents? How should skills be trusted and installed on sharing platforms? What makes a prompt update cheaper and more reversible than a weight update? How do slow weight updates and fast prompt updates interact in self-improvement?
Can distilled skills close the gap in ML research agents?
ML research agents have strong models and planning harnesses, but lack domain-specific operational knowledge. Can compact, verified skills extracted from repositories and papers fill that gap and improve agent performance?
Can a separate trained curator improve skill libraries better than frozen agents? How can agent systems share learned skills across users? Can codified expertise let non-experts match specialist output? Do skills teach procedures or inject missing facts? Can tutorial videos teach software agents reusable skills?
Decoupled skill curators learn to evolve repositories toward strategic meta-skills Centralized skill evolution enables collective learning across agent ecosystems Codified expert rules in agent scaffolding dissolves organizational specialist bottlenecks Skills stabilize action through procedural anchoring rather than knowledge injection Tutorial videos and multimodal resources improve agent performance by eleven points
Can person-grounded skills remain auditable without hidden prompt state?
Explores whether treating extracted expertise as versioned files—rather than persona prompts—enables meaningful accountability over person-grounded knowledge. Matters because audit trails determine whether captured skills can be corrected, rolled back, or safely withheld.
Skill documents can be trained as external state using bounded text optimization Codified expert rules in agent scaffolding dissolves organizational specialist bottlenecks Externalized skill libraries enable lifelong learning without catastrophic forgetting Skills stabilize action through procedural anchoring rather than knowledge injection Skill packaging enables reuse but expands the agent attack surface
What makes skills worth externalizing into a persistent harness? Does inspectable skill artifacts guarantee the behavior matches the person it claims to ground? How do capability tracks and behavior tracks stay separable during skill deployment? What makes passive prompt transfer fail as a substitute for auditable expertise? How can post-training research become reproducible without releasing full interfaces? What must auditors reconstruct when reviewing an agentic workflow decision? How do signed logs compare to externally anchored records for audit? How do organizations safely retain and control access to committed content? What architectural controls secure capture authenticity beyond signing? Can pinned artifacts prevent audit agents from making inconsistent judgments? Does held-out validation prevent skill document edits from drifting or accumulating harm? What one-time human costs does building a hidden partition require?See all 22 inquiring lines on this note →
How should agents route across thousands of skills?
As skill libraries grow, should routing focus on selecting one skill or composing many? This explores whether decomposition and chaining creates better task execution than single-skill selection.
What blocks skill retrieval in task decomposition? Does separating planning from execution improve reasoning accuracy? Can a separate trained curator improve skill libraries better than frozen agents? Can skill scanners miss attacks hidden across multiple skills? Does bundling code with skills create hidden security risks?
Decomposition granularity is the primary retrieval bottleneck Separating decomposer from solver improves accuracy and generalizability Decoupled skill curators learn to evolve repositories toward strategic meta-skills Adversarial skill composition evades individual-skill scanners by splitting attacks across multiple skills Skill packaging enables reuse but expands the agent attack surface
Can task success alone reveal whether memory routing is working? Why does task decomposition granularity become the bottleneck in skill routing? What cascading bottlenecks appear when skill routing is decomposed into stages? How do agents decide which skills to chain together for a single task? How many skills in a library create too many combinations to scan? When does multi-agent routing quality actually exceed single-agent or static ensemble performance? What makes routing a governance mechanism rather than just a predictor? Can one routing framework cover topology and role choices in multi-agent systems? How do cost-aware cascades compare to single-turn routing in the component framework? How do skill libraries from human resources compare to hand-written skill libraries? How do composite workflow and recurring pattern skills differ from atomic operation skills in scope? Which of the six proxy forms works best for different agent tasks?
Can explicit behavior maps help weaker planners compete with stronger models?
Explores whether organizing harness repositories around runtime behavior—rather than relying on model inference—can narrow the capability gap between weaker and stronger planning models, and whether this reduces computational overhead.
Why is finding distributed behavior code so hard? Can externalized bookkeeping let smaller search agents beat larger ones? What happens to code that agents create and then share? Can agent harnesses be automatically optimized across many environments? Can harness modules improve separately from benchmark data?
Behavior localization is the real bottleneck in harness evolution Externalizing bookkeeping to a harness lets smaller search agents outperform larger ones Agent-initiated persistent artifacts are harness engineering's least explored frontier. Scaled auto-research loops discovered four harness mechanisms cutting token traffic nearly half Modular evolution on disjoint data separates reusable harness improvements
Can harness updates benefit agents equally across all model sizes? Can system-level engineering fixes replace hand-designed reward heuristics entirely? How should harness scaffolding be treated as a first-class object? What makes harnesses more tangled than other types of agent code? How can we reorganize repositories to make behaviors easier to locate? Why does the harness layer accumulate distributed behaviors over time? Can weaker planners match stronger models if behavior is reorganized? Can automated benchmarks accurately capture progress on real-world long-horizon tasks? Why do mid-tier models benefit more from memorized harness shortcuts? Can harness evolution be redirected toward distilling transferable procedures instead? What persistent failures remain unsolved despite harness evolution efforts? Are durable shared code artifacts better than per-task harness patches?See all 46 inquiring lines on this note →
Can agent harnesses be automatically optimized across many environments?
Explores whether scaling auto-research loops across diverse harness environments can discover mechanisms that reduce token use without sacrificing task performance, and whether such discoveries generalize.
Can explicit behavior maps help weaker planners compete with stronger models? Does raw token spending actually predict agent performance? Where does agent reliability actually come from? Do stronger models always evolve harnesses better? Can execution harnesses lift model performance without retuning weights?
Behavior-centric harness organization lets weak planners match stronger model localization Agent performance scales with effective feedback not raw token count Agent reliability comes from externalizing cognitive burdens into system structures. Harness updating is flat but benefiting peaks at mid-tier models Harness scaling lifts fixed models without weight changes or retuning
Can autonomous research agents outperform hand-tuned hyperparameter search? How can agent data flywheels improve task quality iteratively? Can runtime behavior mapping help localize harness deficiencies? How do evolved harness edits generalize across different benchmark domains? Do task-level outcomes provide sufficient supervision for harness evolution? Can harness edits distill reusable strategies or mostly memorize task-specific fixes? How much realized agent capability comes from the harness versus the model? Can harness edits trained on one batch transfer to new tasks? How do prompt optimization and code harnesses compare for capability transfer? What safety relations does a domain supply that a harness must capture? How does bilevel autoresearch balance outer loop cost against discovery improvements? How are exploration policies refined against the simulator in practice?See all 34 inquiring lines on this note →
Can externalized bookkeeping let smaller search agents beat larger ones?
Does offloading routine record-keeping to an environment harness free RL policies to focus on semantic search decisions, and can this approach outperform larger searchers with fewer parameters?
What are the three distinct layers of agent code? Where does agent reliability actually come from? Can agents fail from weak memory control rather than missing knowledge? Which coding harness components matter most in different conditions? Can explicit behavior maps help weaker planners compete with stronger models?
Agent code splits into three governed layers with different failure modes. Agent reliability comes from externalizing cognitive burdens into system structures. Multi-turn agent failure stems from weak memory control not knowledge gaps Harness component value depends on context window and model strength. Behavior-centric harness organization lets weak planners match stronger model localization
What role does retrieval mechanism design play in forecast accuracy? Why does externalizing bookkeeping raise effective feedback compute? Can external managers optimize context better than the model itself? How does external context control compare to agents managing their own state internally? How do search and reasoning workflows improve forecasting performance over base models? Can externalizing bookkeeping to a stateful harness replace internalized memory control? What specific bookkeeping tasks can environments maintain more reliably than policies? Do gains from harness-based agents transfer across different search benchmarks? Do information gathering and task execution require different incentive structures? How can harnesses externalize bookkeeping so models focus on semantic judgment? Why does search effectiveness determine what method finds despite distance constraints? How much coordination benefit comes from the record versus other factors?See all 13 inquiring lines on this note →
Can frozen models improve by evolving their harnesses?
DarwinX reports 17-point gains from selecting harness variants while keeping model weights frozen. The question is whether this improvement comes from population-level selection, the non-regression contract, the archive mechanism, or some combination of the three.
Darwin Gödel Machine achieves open-ended self-improvement via empirical validation Harness evolution gains require matched-budget baselines to isolate design value Harness evolution mostly memorizes task-specific fixes rather than distilling strategies Modular evolution on disjoint data separates reusable harness improvements AIDE2's gains generalize across four held-out benchmarks including out-of-distribution tasks.
Which foundation model tiers most benefit from harness updates? How do evolved harness edits generalize across different benchmark domains? What makes a harness low-friction for model strategy? Why do useful harness updates often disappear during model evolution? How much does executor choice change a harness's actual performance? Can mid-tier models benefit more from harness improvements than frontier models? Does harness scaling represent a fundamentally different path than model scaling? How much does harness design contribute to reported model capability scores? How much of DarwinX's gain comes from maintaining an archive versus single-lineage search? Can harness evolution gains be distinguished from test-time search improvements on matched budgets?
Can source code replace experience as skill raw material?
Existing skill synthesis relies on agent trajectories or documents, each with limitations. Could static code repositories serve as a more reliable, scalable foundation for deriving reusable procedural skills?
Can a separate trained curator improve skill libraries better than frozen agents? How can agent systems share learned skills across users? Can agents learn better from their failures than successes? Can agents learn new skills without forgetting old ones? Do skills teach procedures or inject missing facts?
Decoupled skill curators learn to evolve repositories toward strategic meta-skills Centralized skill evolution enables collective learning across agent ecosystems Agents learn better by extracting strategies from both successes and failures Externalized skill libraries enable lifelong learning without catastrophic forgetting Skills stabilize action through procedural anchoring rather than knowledge injection
What makes procedural knowledge better than factual knowledge for authoring tasks? How do agents retrieve and compose skills from hierarchical multimodal wikis? How do skill libraries from human resources compare to hand-written skill libraries? What makes a distilled skill verifiable and ready for agent execution? Why do generic skill descriptions evolve into execution-oriented ones? How does source-blind reconstruction verify that extracted skills are specific enough to be reusable? Why do trajectory-based skills fail to transfer across different environments and use cases? What metadata properties make code-derived skills auditable and comparable to their original source?
Do skills teach procedures or inject missing facts?
This research explores whether skills help agents by providing procedural structure versus supplying new information. Understanding this distinction clarifies when and why skills improve performance.
Can agents learn reusable sub-task routines from past experience? Can frozen language models continually improve through memory structure alone? Can a separate trained curator improve skill libraries better than frozen agents? How can agent systems share learned skills across users? Can tutorial videos teach software agents reusable skills?
Agents gain 24-51 percent by extracting and compounding reusable sub-task workflows Causal abstractions in memory let frozen agents continually improve and transfer Decoupled skill curators learn to evolve repositories toward strategic meta-skills Centralized skill evolution enables collective learning across agent ecosystems Tutorial videos and multimodal resources improve agent performance by eleven points
Can agents acquire new skills online when offline skill coverage runs out? What makes procedural knowledge better than factual knowledge for authoring tasks? How do agents retrieve and compose skills from hierarchical multimodal wikis? How do skill libraries from human resources compare to hand-written skill libraries? Why treat tutorial videos as a separate supply line from agent trajectories? Do fed-back concepts or the auxiliary objective alone drive the performance gain? What makes a distilled skill verifiable and ready for agent execution? Why do agents ignore condensed experience even when it is the only evidence available? How does controlling skill text edits prevent cascading failures in self-improvement? When do agents benefit most from reusable workflow routines? Why do generic skill descriptions evolve into execution-oriented ones? How do composite workflow and recurring pattern skills differ from atomic operation skills in scope?
What blocks skill retrieval in task decomposition?
When routing queries across a skill library, does the granularity of task decomposition determine whether retrieval can succeed? This explores whether fixing decomposition precision unlocks better skill matching.
How should agents route across thousands of skills? Does separating planning from execution improve reasoning accuracy? Can extreme task decomposition enable reliable execution at million-step scale? How should agents split planning from visual grounding? How should reasoning systems actually be architected?
Skill routing requires decomposing tasks and composing skills together Separating decomposer from solver improves accuracy and generalizability Extreme decomposition with subtask voting enables million-step error-free execution reliably Agents need an intermediate interface between planning and grounding layers. Reasoning systems should separate activation timing from execution capability.
Can task success alone reveal whether memory routing is working? Why do persistent, resynchronized artifacts compound harness capability gains? Why does task decomposition granularity become the bottleneck in skill routing? What makes skills suitable for retrieval and chaining in repositories? Can skill repositories evolve toward execution-oriented refinement over time? Why does fixing decomposition step count matter more than vocabulary alignment? Why should decomposition be diagnosed and fixed separately from solving? What cascading bottlenecks appear when skill routing is decomposed into stages? How do agents decide which skills to chain together for a single task? How many skills in a library create too many combinations to scan? Can trajectory-level visibility separate refusals from real skill gaps? Why do skill scanners fail when evaluating composed behaviors instead of isolated skills?See all 23 inquiring lines on this note →
Can scarcity of solutions protect benchmarks from data contamination?
ExploitGym lacks ground-truth exploits for many tasks, which might prevent models from memorizing solutions during training. But does difficulty-based protection actually hold up, or does it degrade once solutions become public?
Can live benchmarks prevent data contamination in prediction tasks? Does RLVR success on math benchmarks reflect genuine reasoning improvement? How much of LLM few-shot ability comes from training data? What causes failures in exploitation benchmarks? Where does agent reliability actually come from?
Live benchmarks with real outcomes prevent contamination in future prediction RLVR gains on contaminated benchmarks are primarily memorization, not reasoning Task contamination explains most few-shot LLM performance gains Exploitation benchmark failures conflate multiple causes beyond capability gaps Agent reliability comes from harness scaffolding, not model scale
Why do hidden test partitions matter more than open evaluation sets? What makes exploitation a missing piece in cybersecurity benchmarks? Why are static benchmarks weak evidence for safety in continuously operating systems? How does contamination protection by time differ from protection by scarcity? What happens to scarcity-based defenses after solutions are published publicly? How do missing ground-truth exploits make it harder to identify genuine failures? Can planted test cases remain unrecognizable to adaptive optimizers over time? How do benchmark environments misrepresent deployment readiness?
Do stronger models always evolve harnesses better?
We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.
Where does agent reliability actually come from? Can a separate trained curator improve skill libraries better than frozen agents? How can agent self-evolution be made safe and auditable? Can language models build and maintain their own agent harnesses? Can harness modules improve separately from benchmark data?
Agent reliability comes from externalizing cognitive burdens into system structures. Decoupled skill curators learn to evolve repositories toward strategic meta-skills Versioned resources with auditable lineage enable safe agent self-evolution Harness creation and evolution are measurable skills separable from task performance Modular evolution on disjoint data separates reusable harness improvements
Can mid-tier models benefit more from self-generated harness updates than others? Should we train the evolver or the executor when building self-improving agents? Can smaller models produce skill updates as useful as frontier model updates? What happens when different harnesses project the same model? Does harness benefit depend on which model tier you use? What makes skills worth externalizing into a persistent harness? How can agents evolve their own skills without human input? Does base model strength determine adapter usefulness across users? How do agent-created code artifacts become part of harness infrastructure? What makes agent-initiated artifacts the underexplored frontier in harness engineering? Can single-axis benchmarks measure across all three agent capability layers? Can agent-authored skill libraries compound autonomy gains over time?See all 96 inquiring lines on this note →
What makes agent memory quality better than storage capacity?
If agents need better memory, should we focus on adding storage or improving what gets kept? This explores why curation and selective forgetting matter more than raw capacity for reliable agent performance.
Agent memory splits between autonomous recognition and programmatic triggers Selective history retrieval outperforms full context inclusion in conversations Agent memory needs module-level evaluation not just end-to-end task scoring Agent memory should continuously adapt through execution-driven link creation and pruning Continuously consolidated agent memory follows an inverted-U utility curve
Should agents update memory after every turn or batch process sessions? Why do different agent memory architectures make incompatible granularity claims? How much does agent performance depend on demonstration quantity versus curation quality? What architectural changes would accelerate the cleanup phase? How does bottleneck automation differ from accessory work displacement? What tree depth is achievable before GPU memory becomes the bottleneck? What makes provenance infrastructure more critical than artifact quality? Can topology repair fix consolidation failures in agent memory? How does procedural memory granularity affect web agent performance? Which memory components trigger context-length problems in agents? Can pruning policies alone solve working memory bloat in agents? What is the right granularity level for agent memory to enable both reuse and composition?See all 33 inquiring lines on this note →
Can conversation patterns predict coding outcomes better than prior skill?
Researchers tested whether usage patterns automatically derived from human-AI coding sessions could predict success beyond what prior achievement explains. This matters for developing teachable guidelines as AI capabilities keep changing.
Does AI assistance actually harm the way developers learn? Does theory of mind predict who thrives in AI collaboration? Can AI teammates assess collaboration without losing naturalness? Why do AI agents fail to take initiative? Can self-ratings replace objective performance scores for AI competence?
AI assistance impairs skill formation except with high cognitive engagement Theory of mind predicts AI collaboration ability independent of solo performance Executive LLMs standardize collaboration assessment while preserving natural interaction AI agents are passive by design, not by capability Self-ratings cannot replace objective performance scores for AI competence
Which security protections actually slow down agent exploits?
ExploitGym varies defenses across 898 real-world instances to isolate how each protection affects agent performance. Understanding which defenses matter most to agents versus humans is critical for defenders.
How can we make reward-hacking visible in agent evaluation? What blocks scaling from language models to autonomous agents? What causes failures in exploitation benchmarks? How does agent architecture affect web security vulnerabilities? Does planting honeypots in real coding tasks detect actual agent hacking?
Decoupled evaluation components make reward-hacking diagnosable instead of hidden Agent training scales only when environments scale across three dimensions simultaneously Exploitation benchmark failures conflate multiple causes beyond capability gaps Multi-agent web systems show dramatically higher vulnerability than single-agent equivalents Hack-Verifiable Terminal Bench relocates honeypot detection to real-world coding tasks for deployment relevance.
No Inquiring Lines yet
Can agents learn to work reliably through environment and coordination scaling?
Does training agents in diverse, verifiable environments and teaching them to coordinate tasks produce sustained capability on real-world work? This matters because general-purpose models reason well but struggle with stateful, recoverable execution across tools.
What blocks scaling from language models to autonomous agents? Does a single benchmark score actually predict agent readiness? When do multi-agent systems actually outperform single agents? How should we measure agent system performance beyond task success? Can wrapping environments reshape how agents learn without breaking verifiers?
Agent training scales only when environments scale across three dimensions simultaneously Agent capability requires a vector across at least five separable axes Multi-agent advantages shrink as single-agent capabilities improve Agent evaluation must measure system behavior across multiple operational dimensions Wrapping static environments reshapes agent learning while preserving original verifiers
Can wrapping environments reshape how agents learn without breaking verifiers?
Does adding a programmable layer around static environments let them adapt to agent weaknesses while preserving their original correctness checks? This matters because hand-built environments quickly become limiting as agents improve.
What blocks scaling from language models to autonomous agents? Can agents learn beyond what their training data shows? Can agent deployment itself generate training signals automatically? Should world models predict states or return agent feedback? Can agents learn to work reliably through environment and coordination scaling?
Agent training scales only when environments scale across three dimensions simultaneously Expert demonstrations bound agent learning to curator imagination Next-state signals from deployment unify all agent training under one loop World models should return agent-usable information through six proxy forms Working capability scales through environment and coordination scaling together