Does AI make coders faster? New study raises red flags

A new study challenges the widely held belief that artificial intelligence tools automatically boost workplace productivity, finding that experienced software developers took longer to complete tasks when using AI than when working without it.

Research conducted by nonprofit technology research group Model Evaluation and Threat Research (METR) found that seasoned developers using AI tools completed their assignments about 19% slower than those who did not rely on the technology. The results surprised both the participants and the researchers, given strong expectations that AI would accelerate software development.

The study involved 16 software developers with an average of five years of professional experience. Together, they completed 246 real-world programming tasks drawn from projects they were already working on. For roughly half of the tasks, developers were allowed to use AI tools such as Cursor Pro and Anthropic’s Claude 3.5 and 3.7 Sonnet. For the remaining tasks, they worked without AI assistance.

Before the experiment, developers estimated that AI would cut their task completion time by an average of 24%. Instead, tasks completed with AI took significantly longer than those done independently.

“One of the biggest surprises was how much time developers spent adjusting and fixing AI-generated code,” said Nate Rush, a technical staff member at METR. While AI outputs were often impressive in isolation, developers reported needing substantial effort to clean up, debug, and adapt the code to fit their existing projects.

Participants also lost time crafting prompts, waiting for responses, and reconciling AI suggestions with their own understanding of complex codebases—context the AI tools lacked.

The findings run counter to optimistic forecasts that AI will deliver sweeping productivity gains, including predictions of double-digit boosts to GDP and workforce efficiency. Recent data suggests those gains have yet to materialize at scale. An MIT report published last year found that only 5% of 300 AI deployments led to rapid revenue growth, while a Harvard Business Review survey showed that just 6% of companies fully trust AI to manage core business operations.

Still, the researchers cautioned against drawing broad conclusions. The study involved a small, specialized sample and captured AI performance at a single point in time. The tools were also new to participants, and future iterations could integrate better with developer workflows.

Economists say the findings align with broader evidence that AI delivers uneven productivity benefits. Anders Humlum, an economist at the University of Chicago Booth School of Business, noted that AI may offer diminishing returns for highly skilled workers who already operate efficiently. His own research across 25,000 workers in Denmark found productivity gains of just 3% among AI users.

MIT economist and Nobel laureate Daron Acemoglu has similarly warned that AI’s economic impact may be overstated, estimating that only a small fraction of tasks will see meaningful efficiency gains.

The study’s authors argue that the takeaway is not to abandon AI, but to deploy it more carefully.

“Some of the decisions being made around AI deployment are high-stakes,” Rush said. “Before automating everything, we need better measurements of where these tools genuinely help—and where they don’t.”