Comment by fmbb | Hacker News Reader

Comment by fmbb | Hacker News Reader

Parent Full thread

> This hurts the article, I think. I don't disagree with his point about companies caring too much about stockholders. But this anti-LLM envangelism just comes across as ludditism.

Is there any research out there suggesting LLMs help programmers get stuff done? I can't say I follow the research closely but I have not seen any.

Googling for [ai llm productivity research] and looking at the first page of results I can't find much interesting evidence. One report says users asked to create a basic web server in JS complete the task 55% faster using an LLM. One report measures "perceived productivity". Students given text writing tasks are faster. One report measures productivity as lines of code written. The rest seem to just be projecting theoretical productivity gains.

Has anyone measured any improved productivity?

I can see this report from METR that is actually measuring something: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

> Core Result > > When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.

But surely someone must have also found results showing productivity gains. I assume I was just not looking hard enough.

I am a happy Copilot Pro user since 2021, still.

> Is there any research out there suggesting LLMs help programmers get stuff done?

Any research will be limited by what the researchers control for?

In my personal experience, when I get a well written bug report or hands on detailed logs, my instinct that I feed it to an Agent and let it figure it all out has never disappointed me. It runs in the background while I work on things I instinctively know the Agent wouldn't do a good job of. How did I develop those instincts? By using Agents for like 3 days. These things (especially, for code completion) are ridiculously effective for the programming languages & code bases I work in, if nowhere else.

> Has anyone measured any improved productivity? ... I am a happy Copilot Pro user since 2021, still.

Whether productivity is tanking or not, I will find it incredibly hard to stop using LLMs/Agents just because a metric or three indicates I'd be better off without them. I must note though, it might be too soon to put a mark on productivity as it is a function of how well new technologies are integrated into processes and workflows, which typically happens over decades and not months/years.

The report is definitely worth reading although I don't think the data is strong enough to draw significant conclusions from. I personally think that this should be studied more in depth over a longer time period since the current study had a 50 hour threshold (not exactly but close enough). One thing which is mentioned in the research paper[1] is:

> However, we see positive speedup for the one developer who has more than 50 hours of Cursor experience, so it’s plausible that there is a high skill ceiling for using Cursor, such that developers with significant experience see positive speedup.

[1]: https://arxiv.org/pdf/2507.09089

Reply on news.ycombinator.com