Why did AI coding tools make seasoned open-source developers slower, even though everyone, including the developers themselves, expected them to speed things up?
Why did early-2025 AI tools slow down experienced open-source developers?
This explores why a randomized trial found that AI coding tools made seasoned open-source developers slower, even though they expected to go faster, and what that gap says about how AI fits into expert work.
This explores why AI coding tools from early 2025 made experienced open-source developers slower, not faster, and why nobody saw it coming. The central study is small and careful: 16 developers worked on 246 real tasks in codebases they knew well, and with AI they took 19% longer. Before starting, they had predicted a 24% speedup. Outside experts in economics and machine learning also expected gains Do AI coding tools actually speed up experienced developers?. The researchers point to three causes: people were too optimistic, the AI was unreliable, and these developers already knew their mature codebases so well that the AI had little to offer them.
The unreliability part makes more sense when you look at a much bigger dataset. In Stack Overflow's 2025 survey, 80% of developers say they use AI tools, but only 29% trust the output to be accurate, down from 40% Why do developers keep using AI tools they don't trust?. Their main complaint is code that looks correct but hides small errors. That is where the time goes. For an expert, writing code was rarely the slow part. Reading AI suggestions, checking them, and fixing them becomes a new task that can take longer than writing the code yourself. In a codebase full of unwritten conventions, the expert's own knowledge is the fast path, and the AI keeps pulling them off it.
The forecasting failure may be the more interesting finding. Why would experienced people misjudge their own speed so badly? One line of work describes four effects that make AI-assisted work feel more competent than it is: it's hard to tell who did what (you or the AI), fluent output looks correct, you hand off thinking without noticing, and you can't see how the output was produced. These effects reinforce each other How do AI tools trick users into overestimating their own skills?. Applied to coding, a session full of quick, confident suggestions can feel productive even when the clock says otherwise. This explains why developers keep using tools they don't trust.
There is also a learning side. A separate randomized trial looked at developers learning unfamiliar libraries. AI help weakened their understanding and their debugging skills, but only for people who used it passively. Developers who stayed actively engaged and worked to understand the AI's output scored 65–86% on comprehension quizzes, compared with 24–39% for passive users Does AI assistance actually harm the way developers learn?. Together the two studies suggest a pattern. Experts in familiar code get little from AI and pay the cost of checking its work. Learners in unfamiliar code get speed but can lose understanding. In both cases, how you use the tool matters more than whether you use it.
One thing worth taking from this: the 19% slowdown is a snapshot of early-2025 tools on one kind of work, which is mature codebases handled by their own experts. It is not a general law. The same gap between expected and measured gains shows up elsewhere in the collection. For example, claims that AI could automate AI research and squeeze years of progress into months rest on unproven assumptions, such as whether skill on small, checkable tasks carries over to messy real research Could automated AI research compress years of progress into months?. The developer trial shows why those assumptions need measuring, not just predicting.
Sources 5 notes
A randomized controlled trial of 16 developers on 246 real tasks found completion times increased 19%, despite developers forecasting a 24% speedup beforehand. Experts in economics and ML also overestimated gains; slowdown factors included over-optimism, low AI reliability, and developers' deep familiarity with mature codebases.
Stack Overflow's 2025 survey shows 80% of developers use AI tools while trust in accuracy fell from 40% to 29%. The primary complaint: AI code that looks correct but contains subtle errors, creating a verification burden that erodes confidence faster than usage grows.
Attribution ambiguity, fluency illusion, cognitive outsourcing, and pipeline opacity combine to systematically misattribute AI outputs as user competence. The effect is multiplicative—each mechanism amplifies the others.
A randomized trial of developers learning new libraries showed AI use degraded conceptual understanding and debugging ability. Six interaction patterns emerged: three low-engagement patterns produced quiz scores of 24-39%, while three high-engagement patterns with active comprehension steps achieved 65-86%, suggesting the mechanism matters more than tool presence.
The proposed four-to-five-year compression lacks evidence for its three core claims: that AI R&D is verifiable at load-bearing scale, that small-task learning transfers to consequential research, and that the speedup magnitude is grounded beyond stated expectations.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- How much does AI impact development speed? An enterprise-based randomized controlled trial
- 2025 Stack Overflow Developer Survey: developers remain willing but reluctant to use AI
- How AI Impacts Skill Formation
- We are Changing our Developer Productivity Experiment Design
- The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows
- Ryan Greenblatt – What happens once AI can automate AI research?
- Microsoft New Future of Work Report 2025