Popular Boards
Julia Becker, Nate Rush, Elizabeth A. Barnes | ArXiv.org | (2025)
Key Takeaways
Sample Definition And Size
The study recruited 16 experienced open‑source developers (each with moderate prior AI experience and on average 5 years of experience in the projects they worked on) who completed a total of 246 real tasks drawn from mature repositories. Each task was randomly assigned to either allow or disallow the use of early‑2025 AI tools. ([arxiv.org](https://arxiv.org/abs/2507.09089?utm_source=openai))
Study Type
Randomized controlled trial (RCT), where tasks were randomly assigned to AI‑allowed or AI‑disallowed conditions to measure the causal impact of AI tool usage on developer productivity. ([arxiv.org](https://arxiv.org/abs/2507.09089?utm_source=openai))
Conflicts Of Interest
No conflicts of interest are declared in the abstract or metadata available from the arXiv entry. ([arxiv.org](https://arxiv.org/abs/2507.09089?utm_source=openai))
Results Summary
Key findings: Developers forecasted a 24% reduction in completion time with AI, and post‑study estimated a 20% reduction, but actual results showed a 19% increase in completion time when using AI tools—i.e., AI slowed developers down. Expert forecasts from economics and ML predicted 39% and 38% speedups, respectively, which were contradicted by the observed slowdown. The study also analyzed 20 hypothesized contributing factors grouped into four categories (direct productivity loss, experimental artifact, human performance factors, AI performance limitations), finding evidence that some factors contributed to the slowdown, mixed or no evidence for others, and evidence against several. ([arxiv.org](https://arxiv.org/abs/2507.09089?utm_source=openai))
Abstract
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.
Referenced In
RC Yu
2 months ago
Created: Mar 27, 2026