Using AI might actually slow down experienced devs
- Experienced developers actually spent 19% more time on tasks when using AI, report claims
- Only 44% of AI-generated code was accepted by experienced devs
- Developers still feel that work is easier when using AI
Artificial intelligence might not be as beneficial to experienced developers as it can be to new starters and those who are yet to develop the right skills, new research has claimed.
A new study conducted by Model Evaluation & Threat Research (METR) has suggested not only were the developers less optimistic following the study, but the real results suggest that artificial intelligence actually ended up costing them time.
The study found that, with AI, devs spent less time coding and searching, and more time prompting, waiting for and more important, reviewing, AI output. An estimated 9% of the time went on reviewing and cleaning up AI-generated code, with AI suggestions generally on the right tracks but lacking in detail.
AI doesn’t actually same these developers any time
Observing 16 experienced developers across 246 real tasks on mature open-source projects that they were already familiar with, the researchers analyzed how the developers interacted with popular tools from Cursor Pro and Claude 3.5/3.7.
Before the study, the 16 experienced developers in question expected to reduce task time by 24% when combining their expertise with artificial intelligence. After the study, they reduced their expectations to just 20%, however post-study analysis reveals that AI actually increased task completion time by 19%.
Fewer than 44% of the suggestions were accepted, with a lack of contextual knowledge and large, complex repositories highlighted as contributing factors to developer slowdown. The study also noted that the experienced developers already had high familiarity with the codebases, leaving little room for AI to add any meaningful value.
However, despite the slowdown, many developers continued to use AI tools because the work felt less effortful, making work feel more pleasant even if it wasn’t faster.
“AI capabilities in the wild may be lower than results on commonly used benchmarks may suggest,” the research paper concludes.
You might also like
Experienced developers actually spent 19% more time on tasks when using AI, report claims Only 44% of AI-generated code was accepted by experienced devs Developers still feel that work is easier when using AI Artificial intelligence might not be as beneficial to experienced developers as it can be to new…
