Should AI systems be designed as subservient co-pilots?

"The Rise of the AI Co-Pilot: Lessons for Design from Aviation and Beyond" proposes the "AI co-pilot" as the governing metaphor for how generative AI systems should be designed to work with people. The paper lists five properties the metaphor implies: the AI "operates under human direction and oversight," "can communicate in a fluid, conversational manner," "can be assigned some degree of agency and intrinsic motivation in working toward shared goals," "has a broad scope for problem solving, but also has a specific set of skills that can supplement as well as complement those of the pilot," and "can act as a back-up for the pilot, helping to monitor a situation and taking over tasks as and when needed." The paper is explicit about which property matters most: "The most important point is at the top of the list: the co-pilot is not the pilot."

The reasoning is a hierarchy of responsibility borrowed directly from the flight deck. "The co-pilot is subservient to direction from the pilot. It is the pilot who makes the critical decisions and has ultimate responsibility for flying the aircraft." Authority over the relationship itself stays with the human: "it is the pilot who oversees the co-pilot, assigning and withdrawing the co-pilot's responsibilities." The co-pilot's competence is real but partial — it holds "some workable grounding about the task at hand" that is only "one part of the more comprehensive and expansive understanding" the pilot needs to fly safely. Agency, in this account, is granted by the human and can be revoked, not an intrinsic property of the system.

This gives the vault a named design metaphor to sit alongside its autonomy frameworks. Can we design AI autonomy separately from capability? offers a graduated version of the same idea — the co-pilot's "assigning and withdrawing" of responsibilities is what moves an agent between those levels in practice. Does targeted human oversight beat both full autonomy and exhaustive review? gives empirical support to the co-pilot's backup-monitoring role specifically, showing that confidence-routed handoff beats both constant oversight and full autonomy. Should AI systems stay collaborative rather than fully autonomous? argues for the same human-retains-control structure from a reliability angle rather than a metaphor.

The excerpt is a conceptual introduction and literature synthesis, not a study: it proposes the metaphor and surveys supporting literature by citation number, but reports no interface, no user test, and no evidence that designing to these five properties changes outcomes. The implication it supports is accordingly a design brief rather than a validated result — that AI systems aiming for a healthy human-AI partnership should be legible about which of the five co-pilot properties they implement, and should make the assignment and withdrawal of responsibility an explicit, human-controlled act rather than an emergent side effect of capability.