How should AI feature risk guide the level of human oversight?

GitLab's Pajamas design system, on its page about AI-human interaction (last updated 2026-07-13), proposes scoring the risk of an AI-powered feature as a function of two variables: "the probability of error and impact of that error, where impact is a comparison of potential cost and benefits." Costs can be "financial, emotional, reputational, opportunity, time, compliance, or others." The guideline anchors the scale with three worked examples drawn from GitLab's own product. A low-risk case is "an agent providing explanations about GitLab features or helping debug configuration issues," where cost stays low because "the user typically reviews and validates information before acting on it." A medium-risk case is "a flow that automatically assigns labels or milestones to work items based on content analysis," where an error costs only "time spent correcting the assignments." A high-risk case is "a flow that automatically creates merge requests with security fixes," where an incorrect change "could introduce new security issues or break functionality," with potential cost compounding "in high-risk applications, such as critical infrastructure."

The scoring scheme is presented in service of a broader design stance: "design AI to be collaborative, not autonomous," and "build trust gradually," starting "with small helpful actions" and expanding capabilities "as users gain confidence with the system." Read against that stance, the risk score is not an abstract classification exercise — it is the variable meant to set how much autonomy a given AI feature is allowed, and how much human review or confirmation has to sit in the loop before an action lands. The guideline also tells designers to "consider if the problem could be addressed with pre-defined rules (if this, then that) before adding AI" at all, treating the risk assessment as a gate on whether AI belongs in the flow in the first place, not only on how it should be exposed once chosen.

This gives a design-system vendor's own ex-ante classification to sit alongside When should agents ask before acting on ambiguous commands?, whose Wizard-of-Oz study arrived at a similar low-stakes/high-stakes split empirically, from observed user preference, rather than from a vendor's internal risk calculus. GitLab's three-tier scheme supplies the production-side counterpart: it names the specific product situations — explanations, auto-labeling, auto-generated security-fix merge requests — that a design team maps onto each tier before a feature ships. It also pairs with Can workplace AI risks emerge from interactions alone?: that framework locates risk in the interaction between agent, goal, and environment even when nothing is broken, while GitLab's scale offers one practical proxy — probability of error times cost of that error — for deciding how tightly to gate such interactions.

The excerpt supplies no measurement of actual error rates, costs, or outcomes for GitLab's own features; the three risk tiers are illustrative examples set on a design-system documentation page, not figures validated by a user study or incident data. It also does not specify how the probability and impact terms combine — additively, multiplicatively, or by threshold — into an actual score, so the scheme reads as a qualitative decision heuristic for designers rather than a calculable formula, and it should be treated as GitLab's own prescriptive stance rather than as evidence that this is how AI-feature risk ought to be measured in general.