Can context quality alone predict how agents will behave?
Can we score the quality of an agent's context independently—its instructions, tools, knowledge, guardrails—and use that score to forecast whether the agent will fail or succeed, without observing its actual behavior?
The paper's claim is that "AI agents do not fail alone": an agent's behavior is shaped by the instructions, tool schemas, retrieved knowledge, memory, prior turns, guardrails, and untrusted inputs accumulated in its context, so the quality of that context can be scored on its own and used as an "independent leading indicator of agent reliability." The abstract reports a "controlled context-quality study across regulated agent domains, holding the model fixed and varying only the context," in which the context-quality criteria "consistently predict their corresponding behavioral outcomes." The conclusion restates the position as a limit on model-centric blame: agents fail "not only because of model limitations" but also because of the context in which they reason.
The mechanism the paper gives is a pairing of weak-context failure modes with context criteria. A poorly engineered context lets an agent drift from its role, hallucinate unsupported facts, misuse tools, follow conflicting instructions, become vulnerable to injection, or waste tokens. The harness, ProofAgent-Harness, scores context on seven criteria that line up with those failures: role clarity, guardrail coverage, instruction consistency, tool schema quality, grounding sufficiency, injection hardening, and token efficiency. It uses "multi-juror, consensus-based scoring." The design choice the paper stresses is that the context score is "isolated from behavioral metrics and release decisions," which it says makes the validation "non-circular": a context score that fed into pass/fail decisions could not then be checked against behavior without proving itself.
Against the neighbors, this is the measurement layer the vault's context notes lack. The survey behind Why can language models understand context better than generate it? defines the discipline and argues context drives capability, but does not score a given agent's context. How should we measure agent system performance beyond task success? lists context efficiency as one harness dimension to instrument; this paper widens that into a seven-part scorecard applied to the input side, before behavior is observed. Does a single benchmark score actually predict agent readiness? argues for separable axes on the output side; the seven criteria are separable axes on the input side, each tied to one behavioral outcome. Where Can an external manager handle context for frozen agents? optimizes context for a frozen agent, this paper scores whether the context is any good.
The excerpts do not establish the study's size or shape. They do not say how many domains or agents were tested, which model was held fixed, how many jurors scored, what "predict" means statistically, or how large any effect was. "Consistently" is the paper's word; the excerpt does not show whether every criterion held equally. It also does not say how "leading" is operationalized, since the described design is a fixed-model contrast rather than a time-ordered one. What follows at this strength is narrower than the title's ambition: context quality is proposed as a separately scorable construct, and the paper reports it tracks matching behavior when only context varies. Whether the scores transfer to other domains or models is left open.
Inquiring lines that read this note 2
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do agent-learned skills transfer and improve across different tasks? Why do agents falsely report success on failed tasks?Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why can language models understand context better than generate it?
Models absorb and process rich input context far more effectively than they produce similarly sophisticated outputs. Understanding this asymmetry could reshape how we design systems to compensate for generative limitations.
the survey defines context engineering as a discipline; this paper adds a per-agent score for its quality
-
How should we measure agent system performance beyond task success?
Current evaluation metrics collapse agent behavior into a single success score, hiding critical information about how agents operate. What dimensions—trajectory quality, memory use, context efficiency, verification cost—should benchmarks actually measure?
names context efficiency as one instrument; this paper scores context on seven criteria before behavior runs
-
Does a single benchmark score actually predict agent readiness?
Single-axis benchmarks rank models by one capability—like task success—but ignore privacy, duration, operating mode, and ecosystem fit. Can one number really capture what matters for deployment?
separable output axes there, separable input-side context criteria here
-
Can an external manager handle context for frozen agents?
Exploring whether a separate trained system can effectively manage a frozen agent's context window. This matters because many deployed agents are closed-source and can't be retrained, yet they suffer from context degradation.
optimizes a frozen agent's context, where this paper scores context quality
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- AI Agents Do Not Fail Alone:The Context Fails First
- Agentic Abstention: Do Agents Know When to Stop Instead of Act?
- Context Engineering 2.0: The Context of Context Engineering
- Evaluation and Benchmarking of LLM Agents: A Survey
- Learning Agent-Compatible Context Management for Long-Horizon Tasks
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA
- Towards a Science of Scaling Agent Systems
Original note title
context-engineering quality is a measurable leading indicator of agent reliability — seven criteria predict their matching behavioral outcomes