METR ran a rare randomized trial on AI's impact in real-world dev work.
Result?
Tasks took 20% longer with AI tools. Even though devs felt 20% faster and experts expected +40%.
No hype - big open-source repos, seasoned devs, Claude 3.5–3.7, Cursor Pro, proper metrics & stat sig.
Turns out - AI slows down experienced devs on real projects.
Full study:
https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf