How should AI systems respond when confidence is low?
When an AI feature lacks confidence in its output, what should the interface do? This explores whether low confidence should trigger fallback paths and how to frame uncertainty so users see it as fixable rather than terminal.
GitLab's Pajamas design system states a specific requirement for what happens when an AI feature is uncertain: "When your system is not certain of the user's intent or has low confidence, make sure there is a path forward that does not rely on AI. Explain why the system was not able to provide a recommendation." The guideline extends the same requirement to breakdowns in live interaction: "When agents cannot understand requests or flows encounter errors, provide clear explanations and alternative paths forward." It then splits the recovery behavior by interaction type — "agent conversations should gracefully handle misunderstandings, while flows should offer users the ability to retry, manually intervene, or fall back to manual processes" — treating a conversational agent's repair and an automated flow's repair as needing different mechanics even though both must leave the user an exit.
The reasoning given is that a confidence gap is informative, not just a failure to suppress. The guideline reframes errors explicitly: "Errors are also opportunities to learn more about your user's mental models and improve the system's ability to make recommendations." It then names the design implication directly — "consider designing a feedback mechanism that presents as a cue for adjustment rather than an error state" — so the interface framing of low confidence is meant to read as an invitation to correct the system's understanding, not as the system failing. This sits downstream of the same document's broader principle that AI should acknowledge uncertainty and defer to user expertise when appropriate, and that users should always "maintain final authority over their work."
This adds a specific, prescriptive behavior to How can users verify what an AI agent actually plans to do?: where that insight argues users need a way to check an agent's goals and actions to build common ground, GitLab's guideline specifies what the agent itself owes the user at the moment common ground breaks down — an explanation of the gap, framed as adjustable rather than terminal, plus a route that works even if the AI component is set aside entirely. It also gives a concrete, product-side instance of the same underlying worry as Do LLM agents misinterpret delegated requests more than formal protocols?: both treat the gap between what a system actually understood and what it claims or implies as something that needs an explicit handling mechanism, though GitLab's answer is an interface pattern for the confidence gap rather than a formal protocol for commitments.
The excerpt does not describe how "low confidence" is computed, what threshold triggers the fallback path, or how GitLab validated that framing errors as "a cue for adjustment" changes user behavior relative to framing them as errors; these are presented as design requirements on a documentation page, not as findings from a usability study. It is also GitLab's own prescriptive stance for its own product surfaces (Duo Chat, Amazon Q, custom agents), not a claim that other AI systems follow or should follow the identical recovery split between conversational agents and automated flows.
Inquiring lines that read this note 3
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How should organizations govern autonomy and access for agentic AI systems? Should AI systems make failures transparent so users can contest and adjust?Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How can users verify what an AI agent actually plans to do?
Stating goals to an AI agent isn't enough for mutual understanding. What mechanisms would let users check both the agent's proposed plan and whether it truly grasps the user's goals and constraints?
specifies what the agent owes the user at the moment of breakdown, not just the need to verify plans beforehand
-
Do LLM agents misinterpret delegated requests more than formal protocols?
Natural-language agent communication infers meaning statistically rather than formally, potentially causing agents to overstep their delegated tasks. How often does this lead to miscoordination in practice?
same underlying gap between what a system understood and what it conveys, handled here as an interface pattern rather than a formal protocol
-
How should AI feature risk guide the level of human oversight?
GitLab's Pajamas design system proposes scoring AI risk as error probability times impact cost. The question explores whether this formula can predict how much autonomy an AI feature should be allowed before human review is required.
sibling note from the same source, on gating autonomy before an action runs rather than recovering after confidence drops
-
How should AI interfaces handle the shift from doing to supervising?
Nielsen explores what UI architecture allows users to oversee AI work rather than perform it themselves. This matters because intent-based systems fundamentally change the user's role from operator to supervisor.
Extends A's non-AI fallback into a general direct-manipulation UI layer for manual correction in mature AI interfaces
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- AI-Human Interaction (Pajamas Design System)
- To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineering
- Exploring automation bias in human-AI collaboration: a review and implications for explainable AI
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search
Original note title
GitLab's Pajamas design system says low-confidence AI should explain the gap and leave a path forward that does not depend on AI