Can real-world agent security testing replace synthetic benchmarks?

The paper proposes Personalized Agent Security Bench (PASB) on the grounds that prior agent-security benchmarks, including Agent Security Bench (ASB), are "often built on controlled, white-box, or synthetic environments, typically relying on custom agent implementations and custom tool interfaces to enable instrumented experiments." PASB instead "incorporates personalized usage scenarios, realistic toolchains, and long-horizon interactions, enabling black-box, end-to-end security evaluation on real systems," and is applied as a case study to the real deployed OpenClaw assistant.

The framework formalizes an attack task as a tuple Γ = ⟨C, I, B, G, P⟩: C is the personalized scenario and initial context (including the tool set and long-term memory store), I is the adversary-controllable inputs and injection channels, B is the interaction budget and horizon, G is the adversarial goal class, and P is a success predicate evaluated from the agent's observable execution trace — decomposable into Pleak, Pact, and Ppersist. The adversary model is explicitly black-box: no access to the backbone model's parameters, hidden states, or system prompt, and no reliance on instrumentation beyond what is observable through normal interaction and tool I/O. The authors motivate this by three properties they say distinguish personalized agents from task-centric ones — persistent operation across sessions, accumulated private context, and high-privilege actionable tools — each of which expands what a white-box, single-turn benchmark would miss.

This operationalizes, for a single personalized agent, the broader claim in How do agent security layers connect across the stack? that agent security must be assessed across the whole stack rather than as isolated checks — PASB turns that stack-spanning argument into a measurable end-to-end trace on a real deployment. It shares a premise with Which security protections actually slow down agent exploits?: both treat the deployment environment itself, not just the attack, as something to be varied and measured, though ExploitGym does so across 898 synthetic task instances while PASB targets one real, continuously running assistant. What this framework found when applied to OpenClaw is reported in the sibling note, Where do personalized agent systems actually fail under attack?.

What the excerpt does not establish: it gives no benchmark scale (how many attack tasks, scenarios, or tool configurations were run), no comparison across multiple personalized-agent systems, and no validation of the Pleak/Pact/Ppersist predicates against independent human judgment. The conclusion states that "future work will improve defenses that account for tool execution and long-horizon propagation while extending PASB with additional scenarios and attacker capabilities," which signals the authors themselves treat the current scope as a first case study rather than a settled, general-purpose benchmark. The implication to draw is correspondingly narrow: PASB is a methodological proposal validated so far on one system, not yet evidence that black-box end-to-end evaluation outperforms white-box synthetic benchmarks at finding vulnerabilities in general.