Before switching on a new AI system, can a company actually measure whether it's ready — people, training, habits — or only find out afterward?
Can organizations measure sociotechnical capital before deploying new AI systems?
This explores whether a company can assess its 'sociotechnical capital' (the people, processes, coordination habits and oversight routines that let a new AI system actually pay off) before rolling the system out, rather than finding out afterward.
This explores whether organizations can take stock of their non-technical readiness (training, workflows, coordination, oversight) before deploying AI, rather than discovering the gaps after launch. The short answer from the corpus is: partly, and not with a single instrument. No note here offers a ready-made 'sociotechnical capital' score. What the corpus does offer is a set of pieces, plus a clear explanation of why this kind of capital is so hard to see.
The most useful starting point is economic. Brynjolfsson, Rock and Syverson's productivity J-curve Why does AI productivity appear weak when investment is high? argues that the investments that make AI work, such as retraining staff, redesigning processes and building supporting software, mostly don't appear in standard accounting. Early on, productivity looks weak because this capital is being built but not counted. Later it can look inflated, once those hidden assets start producing measurable output. So the thing you'd want to measure before deployment is, almost by definition, the thing current measurement misses. That reframes the question: the problem isn't only a missing tool. The usual metrics are pointed in the wrong direction.
Several notes suggest what a pre-deployment audit would need to look at. Agrawal, Gans and Goldfarb Does AI adoption require rethinking how organizations coordinate? show that AI improves individual tasks while making decisions more variable across people, so readiness depends on how well teams coordinate, not on how skilled each person is. BCG's survey of nearly 12,000 employees Does strategic direction matter more than AI tool access? found that clear strategic direction predicted value better than access to tools, which means strategic clarity is something you can survey for in advance. Evans Does easier tool-building actually solve enterprise adoption problems? adds a factor that's easy to overlook: most workers don't see their own tasks as automatable, so 'can our people recognize where AI fits?' is a capability in its own right. Executives also disagree about whether agentic AI is a tool or a coworker Does agentic AI fit the tool-or-coworker framework?, which suggests readiness includes whether HR and asset-management functions are set up to handle something that is partly both.
On the oversight side, the honest picture is fragmentation. One note asks directly what it would take to measure whether a system's errors stay visible and recoverable How can we measure whether AI errors stay visible and recoverable?. It finds partial measures for individual conditions, such as chain-of-thought disclosure, incident counts and rollback timing, but nothing that covers the whole human-plus-institution system. Humberd and Latham Must AI monitoring precede incentive alignment as agents evolve? add an ordering rule: monitoring capacity has to exist before you can align incentives around autonomous AI. That gives you a concrete pre-deployment check, namely whether you can see what the system is doing yet. Evaluation research points the same way. Judging agents by full interaction trajectories instead of final answers How should we evaluate agent behavior beyond final answers? tests recoverability and coordination, which are organizational properties as much as model properties.
The part you may not have expected to want to know: the AI field has the same blind spot as corporate accounting. Agents win benchmark contests but fail long, real occupational workflows Why do agent benchmarks not predict real economic value?, because benchmarks measure isolated tasks instead of work embedded in an organization. In both cases, what gets measured is the visible, task-level piece, and the surrounding social and procedural layer is exactly what's missing. So measuring sociotechnical capital before deployment is possible only as a collection of proxies: strategic clarity, coordination capacity, task recognition and monitoring readiness. Nobody in this corpus has yet built the integrated instrument.
Sources 9 notes
Brynjolfsson, Rock, and Syverson show that GPT adoption generates large unmeasured complementary investments (training, processes, software) that initially undercount true productivity growth, then overcount it once those assets produce measurable output.
A formal model shows AI raises individual task performance but increases decision variation, creating coordination challenges. Organizations must communicate and restructure to capture synergistic returns across interdependent decisions.
BCG's survey of 11,749 employees found that strategic direction predicts sustained value and satisfaction better than tool availability. Companies redesigning workflows end-to-end captured more value than those deploying tools without organizational direction.
Evans argues that reducing coding friction masks two structural barriers: most workers don't see their own tasks as automatable, and enterprise adoption requires organizational decisions that span departments and timelines—not just technical capability.
A survey of 2,102 executives across 116 countries found 76% view agentic AI as a coworker rather than a tool. This duality creates operational tensions because agentic systems simultaneously substitute and complement labor, requiring both HR and asset management approaches that traditional either-or frameworks cannot accommodate.
Show all 9 sources
Partial instruments exist for individual conditions in isolated settings, but none measures the full socio-technical system the paper identifies as necessary. Visibility has a model-side measure (chain-of-thought disclosure), containment has incident-level counts, and recoverability has rollback timing, yet none bridges all four or captures human-institution factors.
Humberd and Latham argue that as AI becomes more autonomous, monitoring mechanisms must be established first as prerequisites to incentive alignment, because defining desirable AI behaviors is harder than for humans and AI decision-making becomes increasingly incomprehensible over time.
Evaluation of agentic systems shifts evidence from final responses to full interaction sequences, and scoring procedure from correctness alone to process quality, recoverability, coordination, and robustness. This pattern appears across multiple agent benchmarks as a coherent design move.
ALE's analysis of 960 real occupational workflows shows agents excel at abstract contests but fail long-horizon professional tasks. The gap is not model capability but benchmark design—the field optimizes what it measures, and it has measured contests rather than work.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Rise of Industrial AI in America: Microfoundations of the Productivity J-curve(s)
- Putting AI on the Org Chart: Evidence on Delegation and Accountability
- When AI Becomes an Agent of the Firm: Examining the Evolution of AI in Organizations Through an Agency Theory Lens
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- The Emerging Agentic Enterprise: How Leaders Must Navigate a New Age of AI
- When Artificial Intelligence Does Strategy: Learning, Good Times, Lock-in, and Human-Driven Strategic Renewal
- Explaining AI Agents Through Execution Traces
- The GenAI Divide: State of AI in Business 2025