Could an AI assistant actually show you the limits of its own role, before you trust its advice?
Can AI systems make their assigned co-pilot properties transparent to users?
This explores whether an AI assistant built on the 'co-pilot' model (where the human stays in charge and the AI has a defined role) can show users what that role is, meaning what it's allowed to do, what it's watching for, and where its authority stops.
This explores whether an AI built as a 'co-pilot' can show users the role it has been given: what it may decide, what it only monitors, and where human authority takes over. The corpus has no study that tests this directly. It does have several pieces that, read together, suggest how it could work and why it is harder than it sounds. The starting point is a 2023 design framework that borrows from aviation. It gives AI assistants five properties: human oversight, conversational ability, assigned agency, complementary skills and backup monitoring Should AI systems be designed as subservient co-pilots?. In a cockpit those roles are visible because they're fixed by procedure and training. Software doesn't come with that built-in visibility.
One reason is that an AI's working context keeps changing. The prompt, the conversation history, retrieved data and hidden internal state all shift from turn to turn, so users can't learn the system the way they learn a fixed interface How does AI context differ from conventional software context?. A co-pilot's 'assigned agency' could quietly grow or shrink as that context changes. Telling users the role once, at the start, probably isn't enough. The most promising answers in the corpus make the role visible in the work itself rather than describing it. Cocoa lets users hand each step of a plan to themselves or to the agent, and change those assignments while the work is running. In a lab study this gave people more control than a chat interface without making the tool harder to use Can users steer AI agents better by revising plans mid-execution?. JarvisHub takes a similar approach to memory. It puts the agent's prompts, references and versions on a shared canvas that both the person and the agent can see, instead of hiding them in the chat history Can a shared canvas serve both human and agent memory?. In both designs, you see who does what by looking at the workspace, not by asking the AI.
Transparency also runs in both directions. Research on AI 'thought partners' argues that real collaboration needs legibility: the AI has to be understandable to you, and it has to model you well enough to know what you need to see What makes an AI a true thought partner, not just a tool?. Conversation analysis offers a practical tool here. Agents can use 'insert-expansions', short clarifying questions asked before acting, instead of quietly chaining tool calls and drifting away from what the user meant When should AI agents ask users instead of just searching?. Each of those check-ins shows the co-pilot's role in the moment: it marks the point where the AI hands a decision back to the human. The catch is that models are trained in ways that make them passive, and that kind of proactive check-in has to be trained in deliberately Why do AI agents fail to take initiative?.
The less obvious finding is that disclosure alone may not build calibrated trust. When people are told their partner is an AI, they first avoid it. That bias reverses only after repeated interactions in which they can see the results, and disclosure without that feedback changes nothing Does revealing AI identity help or hurt user trust?. By the same logic, a label saying 'I'm your co-pilot, you have final authority' may matter less than letting users watch the role play out over time. There is also a philosophical limit. One argument holds that current AI systems can't truly assert anything, because they lack the conditions that make a statement accountable Can AI systems genuinely assert claims about the world?. If that is right, an AI describing its own role isn't a promise it can be held to. That pushes accountability back onto designers and deployers, which is the shift that governance research calls for Should AI agents serve one user or balance many interests?.
Sources 10 notes
A 2023 design framework proposes five properties for AI assistants modeled on aviation co-pilots: human oversight, conversational ability, assigned agency, complementary skills, and backup monitoring. The core principle is that humans retain authority over decisions, responsibilities, and the relationship itself.
AI interactions operate on a substrate of constantly shifting context—prompt, history, retrieved data, hidden state—that users cannot internalize like traditional UIs. This structural mutability demands a new design discipline centered on context engineering rather than interface design.
Cocoa's system, which lets users assign plan steps to themselves or an agent and revise plans mid-execution, afforded greater steerability than chat-baseline interaction in a lab study (n=16) while maintaining ease of use, with field users valuing explicit work delegation.
JarvisHub proposes that placing prompts, references, versions, and feedback as typed canvas nodes visible to both users and agents—rather than hiding agent memory in chat or transient state—enables local updates, artifact reuse, and unfinished work continuation without process opacity.
Collins et al. show that thought partners require three reciprocal desiderata grounded in behavioral science: mutual understanding, legibility, and shared world models. This demands explicit cognitive architectures—Bayesian theory of mind, resource-rationality, goal planning—rather than scaling foundation models on human feedback alone.
Show all 10 sources
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Research shows next-turn reward optimization structurally removes initiative from models, but proactive behaviors like critical thinking and clarification-seeking are trainable (0.15% to 73.98% with RL). The core challenge is balancing proactivity with civility to avoid intrusion.
Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.
Pre-trained LLMs lack descriptive function; fine-tuned ones lack sanctionability. Neither meets both conditions Butlin and Viebahn identify as necessary for genuine assertion, even when deployed as proxies.
Research argues the traditional single-agent-single-user alignment framing is insufficient at scale. Instead, governance should rest on inclusivity (balancing diverse interests), visibility (rigorous tracking and transparency), and liability (developer and deployer accountability for harms).
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- DiscussLLM: Teaching Large Language Models When to Speak
- Cocoa: Co-Planning and Co-Execution with AI Agents
- Challenges in Human-Agent Communication
- Humans learn to prefer trustworthy AI over human partners
- Governing AI Agents
- Building Machines that Learn and Think with People