When experts hand grunt work to an AI, what new job do they end up doing instead?
How are expert professionals shifting their work when they use LLMs for execution?
This explores what happens to experts' own work once they hand the hands-on 'doing' parts of a task to an LLM: what they give up, what they hold onto, and what new work shows up in its place.
This explores how professionals split their work once an LLM takes over execution. The collection has only one direct study of practitioners on this, and its answer is fairly clear. Experts aren't handing over whole jobs. They split the job in two. In interviews with 12 expert visual data storytellers, almost none treated the LLM as an autonomous author. They gave it execution work like drafting, formatting and producing chart code, and kept decisions about what the story means for themselves. The bigger change was in their own time: their effort moved toward checking the output What tasks do expert data storytellers trust to LLMs?. In practice, the expert's job becomes deciding what should be made and confirming it was made correctly, not making it.
System builders have arrived at the same split on their own. In optimization problems, LLMs stall when asked to juggle constraints and iterate on numbers, but they're good at turning a plain-language problem into formal code. The setup that works limits the model to that translation step and gives the rest to a deterministic solver Should LLMs handle abstraction only in optimization?. FlowMind does something similar for business workflows. The LLM builds a plan by calling vetted APIs and never touches confidential data directly, and a human reviews the plan at a high level Can LLMs generate workflows without touching proprietary data?. In both cases the model gets a narrow, checkable role, and someone (or something) reliable owns the result. The storytellers are doing the same thing by hand.
The evidence also suggests why this caution makes sense. LLMs often produce plans that sound right but don't run. In one study only 12% of GPT-4's plans could be carried out without errors Can large language models actually create executable plans?. Long hand-offs are worse. The DELEGATE-52 study passed documents back and forth through a model 50 times, and accuracy drifted in ways single-turn benchmarks don't catch. Models that scored about the same on short tests had diverged sharply by round 25 Do short benchmarks predict how models perform over long workflows?. Giving the model better editing tools didn't help, because the errors come from its judgment about *what* to change, not from how it makes the change Can better tools fix LLM document editing errors?. That is the same judgment experts are keeping for themselves.
The less obvious point is that the verification work experts are taking on looks a lot like what engineers build into software. LLM Programs wrap a model inside an explicit algorithm that controls the sequence of steps and shows each call only the context it needs Can algorithms control LLM reasoning better than LLMs alone?. Research on agent 'harnesses' (the scaffolding around a model) finds that this scaffolding often decides whether actions are grounded or made up Can you turn an LLM into an agent by just fine-tuning?. So an expert who breaks a task into pieces, delegates each piece, and reviews it before moving on is acting as a human harness for the model.
One gap: the collection has a single interview study of working professionals, so how this shift plays out in other fields (law, medicine, engineering) is extrapolated from system-design research, not observed directly.
Sources 8 notes
Interviews with 12 expert visual data storytellers showed they rarely treat LLMs as autonomous storytellers. Instead, they selectively delegate execution-oriented work while retaining human control over narrative meaning, and shift their own labor toward verification.
LLMs plateau at constraint satisfaction regardless of scale, but excel at natural-language-to-formal-structure translation. The productive architecture restricts LLMs to reading input and emitting solver code, leaving numeric iteration to deterministic solvers.
FlowMind demonstrates that LLMs can generate on-the-fly workflows for spontaneous tasks by orchestrating calls to vetted APIs rather than accessing data directly, eliminating confidentiality risks while maintaining high-level human inspection and feedback.
Only 12% of GPT-4 generated plans are actually executable without errors. LLMs excel at acquiring planning knowledge but fail at the reasoning assembly required to handle subgoal and resource interactions.
DELEGATE-52 evaluated models across 50-round-trip relays and found short-interaction performance does not predict sustained delegation accuracy. Models ranking similarly on single-turn tasks diverged dramatically by relay 25, revealing degradation curves invisible to standard benchmarks.
Show all 8 sources
DELEGATE-52 shows that agentic tool access fails to improve performance on long-horizon document tasks. The degradation mechanism originates upstream in the model's judgment about what to change, not in editing interface limitations.
LLM Programs embed LLMs within explicit algorithms that manage control flow and state, presenting only step-specific context to each LLM call. This information hiding addresses capability and context window limits while treating complex reasoning as modular, debuggable sub-tasks.
Converting LLMs to action-capable systems requires four distinct stages: curating action-environment-user datasets, training for action grounding, integrating agent infrastructure with memory and tools, and rigorous safety evaluation. The surrounding system and harness determine whether actions are grounded or hallucinated.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
- LLMs Corrupt Your Documents When You Delegate
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- Can Large Language Models Reason and Plan?
- Harnessing LLMs Without Surrendering Control: Delegation Boundaries in Visual Data Storytelling Authoring
- Position: LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks