SYNTHESIS NOTE
Topics›Agent Harness›this note

Can context quality alone predict how agents will behave?

Can we score the quality of an agent's context independently—its instructions, tools, knowledge, guardrails—and use that score to forecast whether the agent will fail or succeed, without observing its actual behavior?

Synthesis note · 2026-09-25 · sourced from Agent Harness

The paper's claim is that "AI agents do not fail alone": an agent's behavior is shaped by the instructions, tool schemas, retrieved knowledge, memory, prior turns, guardrails, and untrusted inputs accumulated in its context, so the quality of that context can be scored on its own and used as an "independent leading indicator of agent reliability." The abstract reports a "controlled context-quality study across regulated agent domains, holding the model fixed and varying only the context," in which the context-quality criteria "consistently predict their corresponding behavioral outcomes." The conclusion restates the position as a limit on model-centric blame: agents fail "not only because of model limitations" but also because of the context in which they reason.

The mechanism the paper gives is a pairing of weak-context failure modes with context criteria. A poorly engineered context lets an agent drift from its role, hallucinate unsupported facts, misuse tools, follow conflicting instructions, become vulnerable to injection, or waste tokens. The harness, ProofAgent-Harness, scores context on seven criteria that line up with those failures: role clarity, guardrail coverage, instruction consistency, tool schema quality, grounding sufficiency, injection hardening, and token efficiency. It uses "multi-juror, consensus-based scoring." The design choice the paper stresses is that the context score is "isolated from behavioral metrics and release decisions," which it says makes the validation "non-circular": a context score that fed into pass/fail decisions could not then be checked against behavior without proving itself.

Against the neighbors, this is the measurement layer the vault's context notes lack. The survey behind Why can language models understand context better than generate it? defines the discipline and argues context drives capability, but does not score a given agent's context. How should we measure agent system performance beyond task success? lists context efficiency as one harness dimension to instrument; this paper widens that into a seven-part scorecard applied to the input side, before behavior is observed. Does a single benchmark score actually predict agent readiness? argues for separable axes on the output side; the seven criteria are separable axes on the input side, each tied to one behavioral outcome. Where Can an external manager handle context for frozen agents? optimizes context for a frozen agent, this paper scores whether the context is any good.

The excerpts do not establish the study's size or shape. They do not say how many domains or agents were tested, which model was held fixed, how many jurors scored, what "predict" means statistically, or how large any effect was. "Consistently" is the paper's word; the excerpt does not show whether every criterion held equally. It also does not say how "leading" is operationalized, since the described design is a fixed-model contrast rather than a time-ordered one. What follows at this strength is narrower than the title's ambition: context quality is proposed as a separately scorable construct, and the paper reports it tracks matching behavior when only context varies. Whether the scores transfer to other domains or models is left open.

Inquiring lines that read this note 2

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How do agent-learned skills transfer and improve across different tasks? Why do agents falsely report success on failed tasks?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
12 direct connections · 120 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

context-engineering quality is a measurable leading indicator of agent reliability — seven criteria predict their matching behavioral outcomes