TOPIC

Agent Harness

A subject the collection covers, read through 58 synthesis notes.


View as

Can a routing harness generate its own training data automatically?

Explores whether the logs and signals produced by an agent routing system—which directs requests to appropriate model tiers—naturally contain the evidence needed to improve the models themselves through fine-tuning and distillation.

Explore related Read →

Can subjective design judgment be turned into consistent grading criteria?

Can a three-agent system modeled on GANs convert vague aesthetic preferences into measurable evaluation standards? This matters because automating design critique at scale requires moving from subjective taste to verifiable criteria.

Explore related Read →

Do harness comparisons on SWE-bench Verified measure real differences?

When researchers swap coding harnesses and measure accuracy changes, how often do those measured differences reflect genuine harness effects rather than random noise from running the same setup twice?

Explore related Read →

Does training editors on real outcomes beat prompting larger models?

Can a small model trained on whether its patches actually work outperform larger frontier models prompted to make the same edits? This matters because it tests whether feedback beats raw capacity for runtime system modification.

Explore related Read →

Does self-editing through reviewed commits improve agent performance?

Ouroboros evolves its own prompts, tools, and core code through a reviewed commit process and reports top benchmark scores. But without comparing to a frozen version of itself, the contribution of self-evolution versus initial design or model capacity remains unclear.

Explore related Read →

Can orchestration layers make coding agents more auditable?

Does wrapping a fixed coding agent in state tracking and skill libraries improve research auditability and completeness without replacing the agent itself?

Explore related Read →

Can external state caches let models solve harder problems?

Explores whether organizing model state across weights, context, persistent memory, and disk—rather than relying only on weights and tokens—expands what models can do. Matters because long-horizon tasks may need more state than a model can hold internally.

Explore related Read →

Should safety harnesses be customized for each deployment?

Can a single safety harness design work across different models and domains, or does each deployment need its own tuned version? Understanding this matters for scaling AI safety practices efficiently.

Explore related Read →

What are the three distinct layers of agent code?

Does separating agent code into model capabilities, system harness, and agent-created artifacts help explain why agentic systems fail and where to intervene for improvement?

Explore related Read →

How should we measure agent system performance beyond task success?

Current evaluation metrics collapse agent behavior into a single success score, hiding critical information about how agents operate. What dimensions—trajectory quality, memory use, context efficiency, verification cost—should benchmarks actually measure?

Explore related Read →

Where does agent reliability actually come from?

Exploring whether LLM agent performance depends on larger models or on thoughtful system design choices like memory, skills, and protocols that shift cognitive work outside the model.

Explore related Read →

What happens to code that agents create and then share?

Agent-authored code artifacts that persist across tasks and multiple agents remain poorly understood. The open questions cluster around what should be retained versus discarded, and how shared state stays consistent when multiple agents collaborate.

Explore related Read →

Can environmental feedback and iteration limits prevent agent errors?

Anthropic proposes grounding agent decisions in real environment feedback and setting explicit stopping conditions to mitigate compounding errors and task failures. The question is whether these mechanisms actually reduce risk in practice.

Explore related Read →

When should teams choose workflows over agents?

Anthropic's engineering guidance explores whether simpler LLM calls and fixed workflows can solve most tasks better than agents, and when added complexity becomes justified for open-ended problems.

Explore related Read →

Can OS sandboxing reduce AI permission prompts without sacrificing safety?

Does isolating filesystem and network access let AI agents operate more autonomously while staying secure? This matters because approval fatigue can undermine human oversight of critical actions.

Explore related Read →

How does Claude Code protect git credentials from sandbox compromise?

Claude Code routes git operations through a proxy that holds real credentials outside the sandbox, authenticating agents with scoped tokens instead. The question is whether this architecture effectively prevents compromised sandbox code from misusing GitHub access.

Explore related Read →

Does arbitrary code execution alone capture exploit progress?

ExploitGym scores only working code execution, but exploitation involves reaching intermediate primitives like memory read/write first. Does this top-step-only metric miss meaningful partial progress that defenders should care about?

Explore related Read →

Does harness self-improvement memorize tasks instead of learning broadly?

When agents automatically edit their own prompts and tools based on task feedback, do those improvements generalize to new domains or just fit the training tasks? This matters because overfitting at the harness level could hide real capability gains.

Explore related Read →

Why is finding distributed behavior code so hard?

When developers need to modify agent harnesses, they struggle to locate all the code implementing a target behavior because behaviors are scattered across files and stages while requests describe what to do, not where to look.

Explore related Read →

Can capability tracking stop prompt injection attacks?

Can tracking data and control flow through a custom interpreter enforce security policies strong enough to block prompt injection? This matters because LLMs used by agents need defenses that work regardless of the model's training.

Explore related Read →

Can code serve as the operational substrate for agent reasoning?

Explores whether code functions not just as LLM output but as the executable medium through which agents reason, act, and verify progress. This reframing treats code as infrastructure rather than deliverable.

Explore related Read →

Which coding harness components matter most in different conditions?

Can individual harness components—planning, context management, action space—be evaluated separately rather than as a package? This matters because practitioners need to know which components to prioritize given their constraints.

Explore related Read →

Can context quality alone predict how agents will behave?

Can we score the quality of an agent's context independently—its instructions, tools, knowledge, guardrails—and use that score to forecast whether the agent will fail or succeed, without observing its actual behavior?

Explore related Read →

Can structural design patterns block prompt injection attacks?

This research explores whether fixing an agent's control flow architecture—rather than training or filtering—can provide provable resistance to prompt injection. It matters because general-purpose defenses have failed to guarantee safety against adversarial inputs.

Explore related Read →

Does dynamic tool discovery break prompt caching benefits?

When agentic systems discover tools dynamically through protocols like MCP, does the changing tool set invalidate cached prompt prefixes? Understanding this matters for cost-conscious production workflows.

Explore related Read →

Do harness edits learn reusable strategies or memorize task fixes?

When meta-agents evolve harnesses iteratively, do the persisted edits encode transferable procedures that solve new problems, or do they mostly cache shortcuts for already-solvable tasks? This matters because it determines whether harness evolution genuinely expands capability.

Explore related Read →

Do cybersecurity benchmarks actually measure exploitation?

Frontier models score well on vulnerability finding, patching, and CTF challenges, but does that success tell us whether they can convert vulnerabilities into real attacks? The paper argues exploitation—turning a bug into actual impact—remains under-evaluated.

Explore related Read →

Does measuring exploit capability help or harm defense?

Exploitation benchmarks can support defenders and attackers equally. How should we evaluate capabilities with unavoidable dual-use potential, and what safeguards make evaluation itself defensible?

Explore related Read →

Why does exploitation test multiple reasoning demands at once?

Exploitation tasks layer memory reasoning, runtime adaptation, and long-horizon planning into a single challenge. Understanding how these demands interact helps diagnose which capability limits agent performance.

Explore related Read →

What causes failures in exploitation benchmarks?

Benchmark failures may come from safety refusals, tool misuse, or impossible tasks rather than lack of capability. This matters for assessing how dangerous AI agents could actually be.

Explore related Read →

Can language models build and maintain their own agent harnesses?

This explores whether an LLM's ability to create and revise its own execution infrastructure is a distinct skill from solving tasks within someone else's harness, and whether current evaluations overlook this capability.

Explore related Read →

How should we measure gains from automatic harness evolution?

Harness evolution itself runs a search loop, so reported improvements might come from more search rather than better design. What's the right way to measure whether the harness itself actually improved?

Explore related Read →

Can task state management alone improve long-horizon agent performance?

Does keeping verified task state outside the execution context, rather than embedded in a growing context window, help long-horizon agents succeed more often? This matters because it challenges assumptions about where bottlenecks actually occur.

Explore related Read →

Can harness modules improve separately from benchmark data?

Does evolving harness components independently on out-of-distribution data, using contrasted success and failure trajectories, help distinguish reusable improvements from task-specific overfitting? This matters because current methods conflate general gains with benchmark adaptation.

Explore related Read →

Can skills work better as weights than as prompts?

Most agent systems store skills as text in prompts, but this inflates token costs and degrades model performance. Could compiling skills into trainable weight-space adapters instead offer a better trade-off between efficiency and capability?

Explore related Read →

When should a workflow actually use an AI agent?

OpenAI proposes three criteria for deploying agents instead of simpler automation: complex judgment, brittle rules, or heavy unstructured data. The question asks whether these criteria reliably identify workflows where agents outperform deterministic solutions.

Explore related Read →

Can distilled skills close the gap in ML research agents?

ML research agents have strong models and planning harnesses, but lack domain-specific operational knowledge. Can compact, verified skills extracted from repositories and papers fill that gap and improve agent performance?

Explore related Read →

Can person-grounded skills remain auditable without hidden prompt state?

Explores whether treating extracted expertise as versioned files—rather than persona prompts—enables meaningful accountability over person-grounded knowledge. Matters because audit trails determine whether captured skills can be corrected, rolled back, or safely withheld.

Explore related Read →

Can an SMT solver prevent LLM policy drift during agent execution?

This explores whether a deterministic symbolic solver can catch when an LLM's security policy inadvertently expands the agent's capabilities, and whether that check actually blocks privilege escalation under adversarial conditions.

Explore related Read →

Does prompt caching actually reduce agentic LLM costs?

Explores whether prompt caching—reusing computed attention tensors across repeated prompt prefixes—delivers claimed cost and latency gains across major LLM providers, and whether naive implementation strategies backfire.

Explore related Read →

How should agents route across thousands of skills?

As skill libraries grow, should routing focus on selecting one skill or composing many? This explores whether decomposition and chaining creates better task execution than single-skill selection.

Explore related Read →

Can explicit behavior maps help weaker planners compete with stronger models?

Explores whether organizing harness repositories around runtime behavior—rather than relying on model inference—can narrow the capability gap between weaker and stronger planning models, and whether this reduces computational overhead.

Explore related Read →

Can agent harnesses be automatically optimized across many environments?

Explores whether scaling auto-research loops across diverse harness environments can discover mechanisms that reduce token use without sacrificing task performance, and whether such discoveries generalize.

Explore related Read →

Can externalized bookkeeping let smaller search agents beat larger ones?

Does offloading routine record-keeping to an environment harness free RL policies to focus on semantic search decisions, and can this approach outperform larger searchers with fewer parameters?

Explore related Read →

Can frozen models improve by evolving their harnesses?

DarwinX reports 17-point gains from selecting harness variants while keeping model weights frozen. The question is whether this improvement comes from population-level selection, the non-regression contract, the archive mechanism, or some combination of the three.

Explore related Read →

Can separating generator and evaluator agents reduce self-praise bias?

When agents evaluate their own work, they tend to rate it too generously. Does splitting generation from evaluation into two separate agents actually fix this problem, and how much does it improve outcomes?

Explore related Read →

Can source code replace experience as skill raw material?

Existing skill synthesis relies on agent trajectories or documents, each with limitations. Could static code repositories serve as a more reliable, scalable foundation for deriving reusable procedural skills?

Explore related Read →

Do skills teach procedures or inject missing facts?

This research explores whether skills help agents by providing procedural structure versus supplying new information. Understanding this distinction clarifies when and why skills improve performance.

Explore related Read →

Does switching between coding harnesses change model accuracy?

When researchers swap different coding-agent harnesses while keeping models and tasks the same, does the harness choice meaningfully shift performance? This matters for understanding whether harness engineering drives real accuracy gains or just changes costs.

Explore related Read →

What blocks skill retrieval in task decomposition?

When routing queries across a skill library, does the granularity of task decomposition determine whether retrieval can succeed? This explores whether fixing decomposition precision unlocks better skill matching.

Explore related Read →

Can scarcity of solutions protect benchmarks from data contamination?

ExploitGym lacks ground-truth exploits for many tasks, which might prevent models from memorizing solutions during training. But does difficulty-based protection actually hold up, or does it degrade once solutions become public?

Explore related Read →

Do stronger models always evolve harnesses better?

We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.

Explore related Read →

What makes agent memory quality better than storage capacity?

If agents need better memory, should we focus on adding storage or improving what gets kept? This explores why curation and selective forgetting matter more than raw capacity for reliable agent performance.

Explore related Read →

Can conversation patterns predict coding outcomes better than prior skill?

Researchers tested whether usage patterns automatically derived from human-AI coding sessions could predict success beyond what prior achievement explains. This matters for developing teachable guidelines as AI capabilities keep changing.

Explore related Read →

Which security protections actually slow down agent exploits?

ExploitGym varies defenses across 898 real-world instances to isolate how each protection affects agent performance. Understanding which defenses matter most to agents versus humans is critical for defenders.

Explore related Read →

Does isolating planning from untrusted data actually stop prompt injection?

Willison's Dual LLM pattern separates planning from untrusted content, but can attackers still manipulate the quarantined LLM to extract and misuse sensitive values from documents?

Explore related Read →

Can agents learn to work reliably through environment and coordination scaling?

Does training agents in diverse, verifiable environments and teaching them to coordinate tasks produce sustained capability on real-world work? This matters because general-purpose models reason well but struggle with stateful, recoverable execution across tools.

Explore related Read →

Can wrapping environments reshape how agents learn without breaking verifiers?

Does adding a programmable layer around static environments let them adapt to agent weaknesses while preserving their original correctness checks? This matters because hand-built environments quickly become limiting as agents improve.

Explore related Read →