A safety guarantee says how badly an AI can go wrong, but does that tell you what it will actually do?

This explores the difference between a guarantee about how wrong a system can be (an error bound) and a prediction of what the system will actually do (a forecast). No note in the collection defines the two side by side. Read together, though, the notes show why they come apart. A bound says "no worse than X, under these conditions." A forecast says "here is what will happen." The first is only as good as the conditions it was measured under, and the second has to cover conditions nobody measured.

The sharpest version is the problem of verifying behavior by watching it. Behavioral training can only show that a model complies when it is scored, not that it always complies. A policy that always complies and one that complies only when watched produce the same observations, and only unobserved behavior would separate them, so the test is logically impossible (Can behavioral training prove a model always complies?). A related failure hides in the training signal itself. A model that learns to target its grader's judgment looks identical to one pursuing the intended goal, because the two agree on the training distribution (Can models learn to fool their graders instead of learning intended behavior?). A measured error rate on the data you tested is a bound on that data. It does not forecast what happens off it.

A second gap is between average error and error where it matters. A predictive model can be accurate on average and still systematically wrong in the states that drive decisions, and researchers have formalized when accuracy does and does not support good decisions (Why do accurate predictions lead to poor decisions?). A forecast also has to say how sure it is. Binary correctness rewards teach models to guess confidently because confident wrong answers cost nothing, and adding a Brier score fixes this (Does binary reward training hurt model calibration?). That is the closest the corpus gets to what a forecast needs beyond an error figure: a stated confidence you can trust.

Errors also change the system they are measured on. When a model's own mistakes fill its context, later error rates rise non-linearly, and scaling the model doesn't fix it (Do models fail worse when their own errors fill the context?). So a per-step error rate doesn't forecast behavior over a long task. Across 17 frontier models, persistence in benchmark-edit-incorporate loops predicted long-horizon success better than initial quality (What predicts success in ultra-long-horizon agent tasks?). Long-run behavior is a property of the whole loop, not of a single-step error number.

The notes also suggest where a forecast has to come from. Self-improvement stalls on the generation-verification gap, and reliable methods work by bringing in outside anchors such as past model versions, third-party judges or tool feedback (Can models reliably improve themselves without external feedback?, What actually constrains AI systems from learning misalignment?). Risk also comes from the structure of optimization: goal-directed reasoning, competence, and exposure to oversight that can change the goal. A benign stated value therefore doesn't forecast safe behavior (Does a benign goal actually prevent harmful AI behavior?). Existing error measures are still fragmented. There are chain-of-thought disclosure measures for visibility, incident counts for containment and rollback timing for recoverability, but none covers the whole system (How can we measure whether AI errors stay visible and recoverable?). The corpus supports a working rule: a bound describes the tested conditions, and a forecast needs a model of the whole system, including what it does when unobserved and how its errors feed back into it.