Should safety harnesses be customized for each deployment?
Can a single safety harness design work across different models and domains, or does each deployment need its own tuned version? Understanding this matters for scaling AI safety practices efficiently.
The paper's claim is that a safety harness for an LLM agent is not a portable artifact. Its abstract says existing harness designs "are typically built once by experts and applied across heterogeneous models and domains," while "the effective defense is inherently deployment-dependent." It names two failure directions: "A harness strict enough for one model over-blocks another, and a policy general enough to transfer across domains can miss the safety relations of the application." The discussion turns this into a prescription: "A harness should not be copied unchanged across deployments."
The reasoning is a division of labor between two sources of information. The domain "supplies the tool semantics, trusted boundaries, and threat contract that determine the relation to protect and the state it requires." The victim model and runtime "determine its realization: semantic or deterministic, cached or recomputed, and enforced before or after execution." The abstract adds that models differ in "how much external enforcement they need before utility starts to drop," which is why the same check can be right for one model and over-blocking for another. From this the authors conclude that EvoSafeHarness "optimizes a deployment instead of selecting one portable guardrail," and it is presented as a framework that synthesizes a harness "for a frozen model in a target domain." The introduction frames the stakes: the "unit of safety has expanded from a single utterance to an entire action trajectory," and harmful instructions can arrive through indirect prompt injection or direct harmful requests.
This sits close to two notes about what a check must be able to express. The point that a harness needs to know "what history they must remember" is the trajectory thesis in Can step-by-step approval miss harmful behavior patterns?, and it echoes the demand for state in Can stateless checks ever catch sequence-level constraint violations?. Those notes say what a check has to carry; this paper adds that which check, run at which point, is right depends on the deployment. Why does prompt hardening work for single agents but not multi-agent systems? is a consistent case from another paper, where one defense behaved differently across system shapes, though that variable is system architecture rather than model or domain. Which security protections actually slow down agent exploits? treats protections as a variable to control in evaluation; here the harness itself becomes the thing searched over per deployment.
The excerpt reports no results. It does not show that synthesized harnesses outperform expert-built or generic ones, how the optimization runs, what "utility" is measured against, or how much of the gain comes from tuning to the model versus the domain. The claims about over-blocking and missed safety relations are stated as premises, not demonstrated. The safe reading is a design argument: when a harness result is reported, the model and domain it was tuned on are part of the result, and transfer to another pair should be treated as untested.
Inquiring lines that read this note 12
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does harness optimization generalize across different model architectures and domains?- Which foundation model tiers most benefit from harness updates?
- Can runtime behavior mapping help localize harness deficiencies?
- How do evolved harness edits generalize across different benchmark domains?
- What safety relations does a domain supply that a harness must capture?
- What makes a harness low-friction for model strategy?
- Which domains see models exceed human harness design quality?
- Why do useful harness updates often disappear during model evolution?
- How much does executor choice change a harness's actual performance?
- Does harness scaling represent a fundamentally different path than model scaling?
- How much does harness design contribute to reported model capability scores?
- Can weaker models benefit equally from harness updates as stronger ones?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can step-by-step approval miss harmful behavior patterns?
If each action an agent takes passes its individual safety check, can the overall sequence still violate system constraints? This matters because per-action inspection may miss harms that emerge only across time or composition.
the trajectory framing that explains why a harness must remember history
-
Can stateless checks ever catch sequence-level constraint violations?
Explores whether per-action guardrails can express constraints that depend on history, and what structural limits prevent stateless checks from reasoning about composed behavior over time.
the stateless-check limit; this paper adds that the right check varies by deployment
-
Why does prompt hardening work for single agents but not multi-agent systems?
Prompt hardening reduced payload exposure by 40–75% in single-agent systems but failed entirely in multi-agent ones. The gap may reveal how task decomposition breaks the contextual awareness needed for defenses to activate.
one defense working in one system shape and not another, consistent with deployment dependence
-
Which security protections actually slow down agent exploits?
ExploitGym varies defenses across 898 real-world instances to isolate how each protection affects agent performance. Understanding which defenses matter most to agents versus humans is critical for defenders.
protections treated as a variable in evaluation rather than a fixed choice
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- Rethinking the Evaluation of Harness Evolution for Agents
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
- Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA
Original note title
a safety harness should be optimized for each deployment rather than copied — the domain fixes what to protect while the model and runtime fix how