When humans and AI work together at a company, which way of studying that teamwork actually tells you the truth?
What research methods best capture human-AI collaboration in real organizations?
This explores which ways of studying people and AI working together (surveys, experiments, benchmarks, formal models, usage data) actually show what happens inside real workplaces, and what each one misses.
This explores which research approaches show what really happens when people and AI work together inside organizations. The short answer from the corpus is that no single method captures it. Each one sees a different layer, and the most interesting findings appear where those layers disagree.
Start with the methods that measure the AI on its own. Simulated-workplace benchmarks like TheAgentCompany find that leading agents finish only about 30% of realistic office tasks. They fail most often at social interaction, navigating professional software, and domain knowledge (Why do AI agents fail at workplace social interaction?). That tells you how capable the agent is, but nothing about the humans around it. Usage-based research takes the opposite view and tracks where workers have actually handed tasks to AI inside structured workflows. It finds that delegation clusters in information-heavy jobs and follows technical capability, not chatbot popularity. That pattern differs from what classic automation forecasts predicted (Where have workers actually delegated tasks to AI?). Taken together, the two say that what agents can do and what people choose to give them are separate questions, and you need both kinds of data.
The controlled experiments are humbler and more revealing. One study varied only how a human and an agent communicated in a shared task, with 16 participants. That one change shifted trust and awareness of what the other was doing, in the same patterns decades of research found for human-human teams (How do communication modalities shape human-agent collaboration patterns?). The lesson about method: borrow theories from established teamwork research and change one variable at a time, rather than just watching. System-building research works in a similar way. Magentic-UI's designers admit nobody knows the 'right' moment for an agent to ask for help. Instead they spread that decision across six touchpoints, such as planning together, action guards, and verification (When should human-agent systems ask for human help?). Sometimes the most useful finding is an unsolvable problem, plus a record of how people work around it. AI can even become part of the method itself: an LLM teammate can steer a natural conversation toward observable evidence of collaboration skill, then score it about as reliably as human raters do (Can AI teammates assess collaboration without losing naturalness?).
The less obvious point is that the most important effects only show up at the level of the whole organization, and task-level studies can't see them. A formal economic model shows AI makes each person better at their own task while making decisions across people more varied. That creates coordination problems that only appear when you look at how decisions depend on each other (Does AI adoption require rethinking how organizations coordinate?). Puranam goes further and treats every human-AI pair as a small organization. The questions then become who does which kind of task and how feedback reaches each side (How should humans and AI systems divide decision-making tasks?). Executive surveys add a layer of perception: 76% of 2,102 leaders see agentic AI as a coworker rather than a tool, which strains HR and asset-management practices built for one or the other (Does agentic AI fit the tool-or-coworker framework?). Evans points to a blind spot most methods share. Workers often don't recognize their own tasks as automatable, so research that only studies tasks people already handed over will miss everything they never thought to hand over (Does easier tool-building actually solve enterprise adoption problems?).
One honest gap: the corpus has very little long-term, on-site fieldwork, such as ethnographies or case studies that follow a team for months. That matters because what makes an AI feel like a colleague, according to What makes an AI system feel like a colleague rather than a chatbot?, is persistence: memory, reusable procedures, and continuity across tasks. Those qualities only show up over time, and a one-off benchmark or lab session can't capture them. Right now the best picture comes from layering methods. Benchmarks show what agents can do, usage data shows what people delegate, experiments show how people respond, and organizational models show what breaks when it all scales up.
Sources 10 notes
TheAgentCompany benchmark shows leading agents achieve 30% task completion in a simulated workplace. Social interaction, professional UI navigation, and domain-specific knowledge are the three primary failure modes, with multi-turn task performance consistently dropping to 35% across enterprise settings.
Workers have committed AI tasks to structured workflows primarily in information-intensive occupations, following technical capability more than conversational LLM adoption. This gradient differs sharply from routine-task automation predictions and wage patterns reverse at advanced degree levels.
Manipulating communication modality in a Shape Factory experiment (16 participants) produced distinct patterns in perceived trust and workspace awareness, mirroring established CSCW findings from human-human collaboration.
Magentic-UI identifies co-planning, co-tasking, action guards, verification, memory, and multitasking as mechanisms that work around the lack of ground truth for optimal deferral timing. Rather than solving the timing problem directly, these mechanisms distribute decision-making across multiple touchpoints.
An LLM-based approach allows students to collaborate with AI teammates in human-like conversation while the system steers toward observable evidence of skill proficiency. The same LLM can also score the interaction against a rubric with inter-rater agreement matching human performance.
Show all 10 sources
A formal model shows AI raises individual task performance but increases decision variation, creating coordination challenges. Organizations must communicate and restructure to capture synergistic returns across interdependent decisions.
Puranam frames human-AI pairs as multi-agent organizations requiring design choices about task division and learning configuration. He distinguishes three task types and shows learning mechanisms produce different performance outcomes based on feedback structure and decision architecture.
A survey of 2,102 executives across 116 countries found 76% view agentic AI as a coworker rather than a tool. This duality creates operational tensions because agentic systems simultaneously substitute and complement labor, requiring both HR and asset management approaches that traditional either-or frameworks cannot accommodate.
Evans argues that reducing coding friction masks two structural barriers: most workers don't see their own tasks as automatable, and enterprise adoption requires organizational decisions that span departments and timelines—not just technical capability.
Research shows the chatbot-to-colleague shift depends on state persistence, bounded memory, reusable procedures, and task closure—design properties of the system architecture. Larger models alone produce transcripts that disappear; colleagues accumulate experience and maintain workspace continuity across tasks.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Putting AI on the Org Chart: Evidence on Delegation and Accountability
- Unraveling Human-AI Teaming: A Review and Outlook
- When Artificial Intelligence Does Strategy: Learning, Good Times, Lock-in, and Human-Driven Strategic Renewal
- The Rise of Industrial AI in America: Microfoundations of the Productivity J-curve(s)
- How Organizations Use AI: Evidence from ChatGPT
- Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
- From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineering
- The GenAI Divide: State of AI in Business 2025