Hacker News Reader

Comment by llmslave2 | Hacker News Reader

One thing I find really funny is when AI enthusiasts make claims about agents and their own productivity its always entirely anecdotally based on their own subjective experience, but when others make claims to the contrary suddenly there is some overwhelming burden of proof that has to be reached in order to make any sort of claims regarding the capabilities of AI workflows. So which is it?

View on HN

A while ago someone posted a claim like that on LinkedIn again. And of course there was the usual herd of LinkedIn sheep who were full of compliments and wows about the claim he was making: a 10x speedup of his daily work.

The difference with the zillion others who did the same, is that he attached a link to a live stream where he was going to show his 10x speedup on a real life problem. Credits to him for doing that! So I decided to go have a look.

What I then saw was him struggling for one hour with some simple extension to his project. He didn't manage to finish in the hour what he was planning to. And when I had some thought about how much time it would have cost me by hand, I found it would have taken me just as long.

So I answered him in his LinkedIn thread and asked where the 10x speed up was. What followed was complete denial. It had just been a hick up. Or he could have done other things in parallel while waiting 30 seconds for the AI to answer. Etc etc.

I admit I was sceptic at the start but I honestly had been hoping that my scepticism would be proven wrong. But not.

I'm going to try and be honest with you because I'm where you were at 3 months ago

I honestly don't think there's anything I can say to convince you because from my perspective that's a fools errand and the reason for that has nothing to do with the kind of person either of us are, but what kind of work we're doing and what we're trying to accomplish

The value I've personally been getting which I've been valuing is that it improves my productivity in the specific areas where it's average quality of response as one shot output is better than what I would do myself because it is equivalent to me Googling an answer, reading 2 to 20 posts, consolidating that information together and synthesising an output

And that's not to say that the output is good, that's to say that the cost of trying things as a result is much cheaper

It's still my job to refine, reflect, define and correct the problem, the approach etc

I can say this because it's painfully evident to me when I try and do something in areas where it really is weak and I honestly doubt that the foundation model creators presently know how to improve it

My personal evidence for this is that after several years of tilting those windmills, I'm successfully creating things that I have on and off spent the last decade trying to create successfully and have had difficulty with not because I couldn't do it, but because the cost of change and iteration was so high that after trying a few things and failing, I invariably move to simplifying the problem because solving it is too expensive, I'm now solving a category of those problems now, this for me is different and I really feel it because that sting of persistent failure and dread of trying is absent now

That's my personal perspective on it, sorry it's so anecdotal :)

>The value I've personally been getting which I've been valuing is that it improves my productivity in the specific areas where it's average quality of response as one shot output is better than what I would do myself because it is equivalent to me Googling an answer, reading 2 to 20 posts, consolidating that information together and synthesising an output

>And that's not to say that the output is good, that's to say that the cost of trying things as a result is much cheaper

But there's a hidden cost here -- by not doing the reading and reasoning out the result, you have learned nothing and your value has not increased. Perhaps you extended a bit less energy producing this output, but you've taken one more step down the road to atrophy.

Seeing the code that the LLM generates and occasionally asking it to explain has been an effective way to improve my understanding. It's better in some ways than reading documentation or doing tutorials because I'm working on a practical project I'm highly motivated by.

I agree that there is benefit in doing research and reasoning, but in my experience skill acquisition through supervising an LLM has been more efficient because my learning is more focused. The LLM is a weird meld of domain expert/sycophant/scatterbrain but the explanations it gives about the code that it generates are quite educational.

I think there's a potential unstated assumption here, though forgive me if it was made explicit elsewhere and/or I missed it.

LLM-assisted can be with or without code review. The original meaning of "vibe coding" was without, and I absolutely totally agree this rapidly leads to a massive pile of technical debt, having tried this with some left-over credit on a free trial specifically to see what the result would be. Sure, it works, but it's a hell of a mess that will make future development fragile (unless the LLMs improve much faster than I'm expecting) for no good reason.

Before doing that, I used Claude Code the other way, with me doing code reviews to make sure it was still aligned with my ideas of best practices. I'm not going to claim it was perfect, because it did a python backend and web front end for a webcam in one case and simultaneously on a second project a browser-based game engine and example game for that engine and on a third simultaneous project a joke programming language, and I'm not a "real" python dev or "real" web dev or any kind of compiler engineer (last time I touched Yacc before this joke language was 20 years earlier at university). But it produced code I was satisfied I could follow, understand, wasn't terrible, had tests.

I wouldn't let a junior commit blindly without code review and tests because I know what junior code looks like from all the times I've worked with juniors (or gone back to 20 year old projects of my own), but even if I was happy to blindly accept a junior's code, or even if the LLM was senior-quality or lead quality, the reason you're giving here means code review before acceptance is helpful for professional development even when all the devs are at the top of their games.

Yes, but I'm talking about more then code review -- there is a ton of value in discovering all of the ways not to solve a problem. When reading 25 forum posts or whatever in trying to write some function, you're learning more then just the answer. You're picking up a ton of context about how these sorts of problems are solved. If all you're doing is reviewing the output of some code generator, your mental context is not growing in the same way.

I'm curious if you think the same thing was lost with the transition from reading man pages and first-party documentation to going to stackoverflow or google first (at least, I assume the former was more common a couple decades ago)

What was lost in that transition was the required quality of that first-party documentation decreased; generally that first party documentation simply didn't contain enough information, so you needed to determine things empirically or read source code to get more information. I do think the culture of "copy-and-paste from stackoverflow" harmed the general competency of programmers, but having more third-party information available was only a positive thing.

Before 2022 age of modern AI, man pages, SO and Google all were the results of humans, not AI fabrication and hallucination.

A lot was lost then too.

Merely choosing lines to copy and paste from one file of your own code to another is a learning experience for your brain. AI is excellent for removing a lot of grunt work, but that type of work also reinforces your brain even if you think you are learning nothing. Something can still be lost even if AI is merely providing templates or scaffolding. The same can be said of using Google to find examples, though. You should try to come up with the correct function name or parameter list yourself in your head before using a search engine or AI. And that is for the moist simple examples, e.g. "SQL table creation example". These should be things we know off the top of our heads, so we should first try to type it out before we go to look for an answer.

> By not doing the reading and reasoning out the result, you have learned nothing and your value has not increased

AI helps at the margins.

It’s like adding anti-piracy. Some people would simply never have bought the game unless they can pirate it.

There’s a large volume of simple tools, or experimental software that I would simply never had the time to build the traditional way.

I mean you're not wrong

I suppose the way I approach this is, I use libraries which solve problems that I have, that in principle understand, because I know and understand the theory, but in practice I don't know the specific details, because I've not implemented the solution myself

And honestly, it's not my job to solve everything, I've just got to build something useful or that serves my goals

I basically put LLM's into that category, I'm not much of a NIH kinda person, I'm happy to use libraries, including alpha ones on projects if they've been vetted over the range of inputs that I care about, and I'm not going to go into how to do that here, because honestly it's not that exciting, but there's very standard boring ways to produce good guarantees about it's behaviour, so as long as I've done that, I'm pretty happy

So I suppose what I'm saying is that isn't a hidden cost to me, it's a pragmatic decision I made that I was happy with the trade off :)

When I want to learn, and believe me I do now and again, I'll focus on that there :)

> I use libraries

> I basically put LLM's into that category

That says a lot to be sure.

Seeing as you've chosen to be ambiguous, I'll interpret your comment positively :)

Otherwise feel free to put forward a criticism

Example for me: I am primarily a web dev today. I needed some kuberntes stuff setup. Usually that’s 4 hours of google and guess and check. Claude did it better in 15 minutes.

Even if all it does is speed up the stuff i suck at, that’s plenty. Oh boy docker builds, saves my bacon there too.

And you learned nothing and have no clue if what it spit out is good or not.

How can you even assume what it did is "better" if you have no knowledge of kubernetes in the first place? It's mere hope.

Sure it gets you somewhere but you learned nothing in the way and now depend on the LLM to maintain it forever given you don't want to learn the skill.

I use LLMs to help verify my work and it can sometimes spot something I missed (more often it doesn't but it's at least something). I also automate some boring stuff like creating more variations of some tests, but even then I almost always have to read its output line by line to make sure the tests aren't completely bogus. Thinking about it now it's likely better if I just ask for what scenarios could be missing, because when they write it, they screw it up in subtle ways.

It does save me some time in certain tasks like writing some Ansible, but I have to know/understand Ansible to be confident in any of it.

These "speedups" are mostly short term gains in sacrifice for long term gains. Maybe you don't care about the long term and that's fine. But if you do, you'll regret it sooner or later.

My theory is that AI is so popular because mediocrity is good enough to make money. You see the kind of crap that's built these days (even before LLMs) and it's mostly shit anyways, so whether it's shit built by people or machines, who cares, right?

Unfortunately I do, and I rather we improve the world we live in instead of making it worse for a quick buck.

IDK how or why learning and growing became so unpopular.

> Sure it gets you somewhere but you learned nothing in the way and now depend on the LLM to maintain it forever given you don't want to learn the skill.

The kind of person who would vibe code a bunch of stuff and push it with zero understanding of what it does or how it does it is the kind of person who’s going to ruin the project with garbage and technical debt anyway.

Using an LLM doesn’t mean you shouldn’t look at the results it produces. You should still check it results. You should correct it when it doesn’t meet your standards. You still need to understand it well enough to say “that seems right”. This isn’t about LLMs. This is just about basic care for quality.

But also, I personally don’t care about being an expert at every single thing. I think that is an unachievable dream, and a poor use of individual time and effort. I also pay people to do stuff like maintenance on my car and installing HVAC systems. I want things done well. That doesn’t mean I have to do them or even necessarily be an expert in them.

I notice this already after around of 6 months heavy usage. Skills decline, even information gathering etc

I think it is more accurate to say some skills are declining (or not developing) while a different set of skills are improving (the skill of getting an LLM to produce functional output).

Similar to if someone started writing a lot of C, their assembly coding skills may decline (or at least not develop). I think all higher levels of abstraction will create this effect.

> while a different set of skills are improving (the skill of getting an LLM to produce functional output

Lmaooooo

I agree with both of your points since I use LLMs for things I am not good at and dont give a single poop about. The only things i did with LLMs are three examples from the last two years:

- Some "temporary" tool I built years ago as a pareto-style workaround broke. (As temporary tools do after some years). Its basically a wrapper that calls a bunch of XSLs on a bmecat.xml every 3-6 months. I did not care to learn XSL back then and I dont care to do it now. Its arcane and non-universal - some stuff only works with certain XSL processors. I asked the LLM to fix stuff 20 times and eventually it got it. Probably got that stuff off my back another couple years.

- Some third party tool we use has a timer feature that has a bug where it sets a cookie everytime you see the timer once per timer (for whatever reason... the timers are set to end a certain time and there is no reason to attach it to a user). The cookies have a life time of one year. We run time limited promotions twice a week so that means two cookies a week for no reason. Eventually our WAF got triggered because it has a rule to block requests when headers are crazy long - which they were because cookies. I asked an LLM to give me a script that clears the cookie when its older than 7 days because I remember the last time i hacked together cookie stuff it also felt very "wtf" in a javascript kinda way and I did not care to relive that pain. This was in place until the third party tool fixed the cookie lifetime for some weeks.

- We list products on a marketplace. The marketplace has their own category system. We have our own category system. Frankly theirs kinda suck for our use case because it lumps a lot of stuff together, but we needed to "translate" the categories anyway. So I exported all unique "breadcrumbs" we have and gave that + the categories from the marketplace to an LLM one by one by looping through the list. I then had an apprentice from another dept. that has vastly more product knowledge than me look over that list in a day. Alternative would have been to have said apprentice do that stuff by hand, which is a task I would have personally HATED so I tried to lessen the burden for them.

All these examples are free tier in whatever I used.

We also use a vector search at work. 300,000 Products with weekly updates of the vector db.

We pay 250€ / mo for all of the qdrant instances across all environments and like 5-10 € in openai tokens. And we can easily switch whatever embedding model we use at anytime. We can even selfhost a model.

No I agree with you, there are area's where AI is helping amazingly. Every now and then it helps me with some issue as well, which would have cost me hours earlier and now it's done in minutes. E.g. some framework that I'm not that familiar with, or doing the scaffolding for some unit test.

However this is only a small portion of my daily dev work. For most of my work, AI helps me little or not at all. E.g. adding a new feature to a large codebase: forget it. Debugging some production issue: maybe it helps me a little bit to find some code, but that's about it.

And this is what my post was referring to: not that AI doesn't help at all, but to the crazy claims (10x speedup in daily work) that you see all over social media.

[flagged]

It's great in unfamiliar terrain.

It's great when the terrain is unfamiliar to the user but extremely familiar to the LLM. And it's useless in the opposite.

The best programmers are going to be extremely familiar with terrains that are unfamiliar to the LLMs, which is why their views are so negative. These are people working on core parts of complex high performing highly scalable systems, and people with extreme appreciation for the craft of programming and code quality.

But the most productive developers focused on higher level user value and functionality (e.g pumping out full stack apps or features), are more likely to be working with commonly used technologies while also jumping around between technologies as a means to a functionality or UX objective rather than an end of skill development, elegant code, or satisfying curiosity.

I think this explains a lot of the difference in perspectives. LLMs offer value in the latter but not the former.

It's a shame that so many of the people in one context can't empathize with the people in the other.

you haven't contributed much to GitHub since 2022?

*edit unless your commits are elsewhere?

I think people get into a dopamine hit loop with agents and are so high on dopamine because its giving them output that simulates progress that they don't see reality about where they are at. It is SO DAMN GOOD AT OUTPUT. Agents love to output, it is very easy to think its inventing physics.

Obviously my subjective experience

Ironic that I’m going to give another anecdotal experience here, but I’ve noticed this myself too. I catch myself trying to keep on prompting after an llm has not been able to solve some problem in a specific way. While I can probably do it faster at that point if I switch to doing it fully myself. Maybe because the llm output feels like its ‘almost there’, or some sunken cost fallacy.

Not saying this is you, but another way to look at it is that engaging in that process is training you (again, not you, the user) -- the way you get results is by asking the chat bot, so that's what you try first. You don't need sunk cost or gambling mechanics, it's just simple conditioning.

Press lever --> pellet.

Want pellet? --> press lever.

Pressed lever but no pellet? --> press lever.

> I think people get into a dopamine hit loop

I also think that's the case, but I'm open to the idea that there are people that are really really good at this and maybe they are indeed 10x.

My experience is that for SOME tasks LLMs help a lot, but overall nowhere near 10x.

Consistently it's probably.... ~1X.

The difference is I procrastinate a lot and LLMs actually help me not procrastinate BECAUSE of that dopamine kick and I'm confident I will figure it out with an LLM.

I'm sure there are many people who got to a conclusion on their to-do projects with the help of LLMs and without them, because of procrastination or whatever, they would not have had a chance to.

It doesn't mean they're now rich, because most projects won't make you rich or make you any money regardless if you finish them or not

You nailed it - like posting on social media and getting dopamine hits as you get likes and comments. Maybe that's what has got all these vibe coders hooked.

> What I then saw was him struggling for one hour with some simple extension to his project. He didn't manage to finish in the hour what he was planning to. And when I had some thought about how much time it would have cost me by hand, I found it would have taken me just as long.

For all who are doing that, what is the experience of coding in a livestream? It is something I never attempted, the simple idea makes me feel uncomfortable. A good portion of my coding would be rather cringe, like spending way too long on a stupid copy-paste or sign error that my audience would have noticed right away. On the other hand, sometimes, I am really fast because everything is in my head, but then I would probably lose everyone. I am impressed when looking at live coders by how fluid it looks compared to my own work, maybe there is a rubber duck effect at work here.

All this to say that I don't know how working solo compares to a livestream. It is more or less efficient, maybe it doesn't matter that much when you get used to it.

Have done it, never enough of an audience to be totally humiliated. It's never going to be more efficient.

But as for your cringe issue that the audience noticed, one could see that to be a benefit -- prefer to have someone say e.g. "you typed `Normalise` (with an 's') again, C++ is written in U.S. English, don't you know / learn to spell, you slime" upfront than waiting for compiler to tell you that `Normalise` doesn't exist, maybe?

I suspect livestream coding, like programming competition coding and whiteboard coding for interviews, is a separate skill that's fairly well correlated with being able to solve useful problems, but it is not the same thing. You can be an excellent problem solver without being good at doing so while being watched, under time pressure.

I feel like I've been incredibly productive with AI assisted programming over the past few weeks, but it's hard to know what folks' baselines are. So in the interest of transparency, I pushed it all up to sourcehut and added Co-Authored-By footers to the AI-assisted commits (almost all of them).

Everything is out there to inspect, including the facts that I:

- was going 12-18 hours per day

- stayed up way too late some nights

- churned a lot (+91,034 -39,257 lines)

- made a lot of code (30,637 code lines, 11,072 comment lines, plus 4,997 lines of markdown)

- ended up with (IMO) pretty good quality Ruby (and unknown quality Rust).

This is all just from the first commit to v0.8.0. https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0

What do you think: is this fast, or am I just as silly as the live-streamer?

P.S. - I had an edge here because it was a green-field project and it was not for my job, so I had complete latitude to make decisions.

I don't really know Ruby, so maybe I'm missing something major, but your commit messages seem extremely verbose yet messy (I can't make heads or tails of them) and I'm seeing language like "deprecated" and a stream of "releases" within a period of hours and it just looks a bit like nonsense.

Don't take "nonsense" negatively, please -- I mean it looks like you were having fun, which is certainly to be encouraged.

The commit messages with a Co-Authored-By footer were all generated. I recommend clicking the "tree" link to see the actual code. Specifically:

- README.md explains the basics https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0/item/REA...

- CHANGELOG.md is better than the commit messages, and filtered to only what app devs using the library likely care about: https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0/item/CHA...

- doc/ holds the Markdown documentation, which I heavily reviewed. https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0/item/doc

- lib/ holds the Ruby source code of the library, which I heavily designed and reviewed. https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0/item/lib

- examples/ holds the Ruby source code of some toy apps built with the library. https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0/item/exa...

- bin/ holds a few Ruby scripts & apps to automate some ops (check out announce) https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0/item/bin

- tasks/ holds some more Ruby scripts & apps to automate some ops (most I did not read, but I heavily designed and reviewed bump and terminal_preview) https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0/item/tas...

- ext/ holds the Rust source code of the library, which I did not read most of. https://git.sr.ht/~kerrick/ratatui_ruby/tree/v0.8.0/item/ext

I was having a lot of fun, and part of the reason I took deprecations and releases seriously was because I hoped to encourage adoption. And that I did: https://todo.sr.ht/~kerrick/ratatui_ruby/4 and https://github.com/sidekiq/sidekiq/blob/main/bin/tui

There were such people also here.

Copy-pasting the code would have been faster than their work, and there were several problems with their results. But they were so convinced that their work is quick and flawless, that they post a video recording of it.

Hackernews is dominated by these people

LLM marketers have succeeded at inducing collective delusion

> LLM marketers have succeeded at inducing collective delusion

That's the real trick & one I desperately wish I knew how to copy.

I know there's a connection to Dunning Kruger & I know that there's a dopamine effect of having a responsive artificial minion & there seems to be some of that "secret knowledge" sauce that makes cults & conspiracies so popular (there's also the promise of less effort for the same or greater productivity).

Add the list grows, I see the popularity, but I doubt I could easily apply all these qualities to anything else.

IMO algorithmically generated "social" media feeds combined with the lack of adequate mass-media alternatives have supercharged cult recruitment in the last approximately 10 years.

Stupid people in my life have been continually and recklessly joining harebrained cults for the last 5 years.

Really I think it's probably much, much easier to start a cult these days than it has ever been. Good news for tech company founders I guess, bad news for American culture, American society, and the American people.

One way to help stop it is to get off social media and stop giving these tech billionaires so much money.

The less people on social media, the less real the network effect is, the less people who join in the first place, the less money the billionaires have to throw hundreds of millions into politics, the less inadvertent cult members.

I've gotten to the point where I just leave my phone at home at this point, and it has been incredibly nice. Before that I deleted most apps that I found to be time wastes, deleted all social media (HN and two small discords are my exception).

It's very nice, I'm less stressed, I feel more in the moment, I respond to my friends when I check my phone every few hours on the speaker in the other room.

I encourage others to try it, add it to your dry January.

And ya know what I ain't doing a lick of? Sending money and reams of data to these billionaires I think are really lame individuals with corrupted moral compasses.

Now it ain't perfect, I'm sure Google's still getting reams of info about me from my old Gmail account that I still use sometimes, and Apple too from a few sources. But... getting closer!

So many folk sit here and recognize the same problems I do, the way it warps your attention, the addictiveness of the handheld devices, the social media echo chambers, the rising influence of misinformation, the lack of clarity between real and fake...

Seems like there's a solution in front of us :-)

> So I answered him in his LinkedIn thread and asked where the 10x speed up was. What followed was complete denial. It had just been a hick up. Or he could have done other things in parallel while waiting 30 seconds for the AI to answer. Etc etc.

So I’ve been playing with LLMs for coding recently, and my experience is that for some things, they are drastically faster. And for some other things, they will just never solve the problem.

Yesterday I had an LLM code up a new feature with comprehensive tests. It wasn’t an extremely complicated feature. It would’ve taken me a day with coding and testing. The LLM did the job in maybe 10 minutes. And then I spent another 45 minutes or so deeply reviewing it, getting it to tweak a few things, update some test comments, etc. So about an hour total. Not quite a 10x speed up, but very significant.

But then I had to integrate this change into another repository to ensure it worked for the real world use case and that ended up being a mess, mostly because I am not an expert in the package management and I was trying to subvert it to use an unpublished package. Debugging this took the better part of the day. For this case, the LLM may be saved me maybe 20% because it did have a couple of tricks that I didn’t know about. But it was certainly not a massive speed up.

So far, I am skeptical that LLM’s will make someone 10x as efficient overall. But that’s largely because not everything is actually coding. Subverting the package management system to do what I want isn’t really coding. Participating in design meetings and writing specs and sending emails and dealing with red tape and approvals is definitely not coding.

But for the actual coding specifically, I wouldn’t be surprised if lots of people are seeing close to 10x for a bunch of their work.

I suspect there's also a good amount of astroturfing happening here as well, making it harder to find the real success stories.

I've noticed a similar trend. There seems to be a lot of babysitting and hand holding involved with vibe-coding. Maybe it can be a game changer for "non-technical founders" stumbling their way through to a product, but if you're capable of writing the code yourself, vibe coding seems like a lot of wasted energy.

Shopify's CEO just posted the other day that he's super productive using the newest AI models and many of the supportive comments responding to his claim were from CEOs of AI startups.

You're supposed to believe in his burgeoning synergy so that one day you may collaborate to push industry leading solutions

Even if this would take two, three hours and a vibe coder, still cheaper then a real developer

Theres too much money, time and infrastructure committed for this to be anything but successful.

Its tougher than a space race or the nuclear bomb race because there are fewer hard tangibles as evidence of success.

I think there is also some FOMO involved. Once people started saying how AI was helping them be more productive, a lot of folks felt that if they didn't do the same, they were lagging behind.

Sounds like someone trying to sell a course or something.

10 times zero is still zero!

Maybe he would have otherwise struggled for 10 hours on that extension.

It's an impossible thing to disprove. Anything you say can be countered by their "secret workflow" they've figured out. If you're not seeing a huge speedup well you're just using it wrong!

The burden of proof is 100% on anyone claiming the productivity gains

I go to meetups and enjoy myself so much; 80% of people are showing how to install 800000000 MCPs on their 92gb macbook pros, new RAG memory, n8n agent flows, super special prompting techniques, secret sauces, killer .md files, special vscode setups and after that they still are not productive vs just vanilla claude code in a git repos. You get people saying 'look I only have to ask xyz... and it does it! magic' ; then you just type in vanilla CC 'do xyz' and it does exactly the same thing, often faster.

This was always the case. People obsessing over keyboards, window managers, emacs setups... always optimizing around the edges of the problem, but this is all taking an incredible amount of their time versus working on real problems.

Yes, the thing they realize much later in life is that perhaps they enjoyed the act of gardening (curating your tools, workflows, etc) much more than farming (being downright focused and productive on the task at hand).

Sadly gardening doesn’t pay the bills!

If I only spend $1000 on hydroponics and 3 weeks tending to a vertical garden I can grow a $1 head of lettuce FOR FREE!

I tried growing lettuce in some cut up plastic bottles at university in soil from the nearby footpath, I think even with the cheap student approach I spent more on the (single pack of) seeds than a fully grown lettuce costs, and only managed about four individual leaves that were only about 1cm by 5cm.

What if I haven’t spent anything,

and I’m making money with lettuce I grew in the woods?

(or, in Anthropic/sama’s backyards)

yep, and I have the same thing, but then I am not going to tell everyone it is super productive for the actual task of farming. I say that I have a hobby farming (which I do) and talk about my tools and my meager yields (which won't make any money if sold). I am not going to say that my workflows are so productive while my neighbour who is a professional farmer just has old crap and just starts and works from 5 am to 9 pm making a living of his land.

I like farming but a lot of the tools are annoying to use so I find myself tinkering with them (gardening in your analogy I guess). It's not that I prefer tinkering in the shop to farming. More that I just have very little patience for tools that don't conform to the ways in which I think about the world.

Same thing happens in music production. If only I had this guitar, or that synth, or these plugins…

Gear Acquisition Syndrome is a very different problem. Even if you haven't cured the issue the new synth was meant to fix, at least you have a new synth.

It's the four hobbies all over again: https://brooker.co.za/blog/2023/04/20/hobbies.html

A better keyboard is a hill I will die on.

I have a fantastic keyboard, but I'm not taking pictures of it, changing the keycaps, posting about it. It's a tool, not a fetish; that's how I differentiate these things.

It's a keyboard attached to an article of clothing you put your head into so the keys drape over your shoulders. You then type, but also end up giving yourself a shoulder massage!

Yes, this happens quite often. So often that I wonder if it is among the symptoms of some psychiatric or neurological disorder.

It's just boredom probably. Obsessing over productivity tools is relatively more productive than say, something completely unrelated to your work.

That perfectly ties with my experience. Just direct prompts, with limited setup and limited context seem to work better or just as well as complex custom GPTs. There are not just diminishing, but inverting returns to complexity in GPTs

limited prompts work well for limited programs, or already well defined and cemented source bases.

once scope creeps up you need the guardrails of a carefully crafted prompt (and pre-prompts, tool hooks, AGENTS files, the whole gambit) -- otherwise it turns into cat wrangling rapidly.

Not in our (30+ year old software company) experience and we have large code bases with a lot of scope creep ; more than ever as we can deliver a lot more for a lot less (lot more revenue / profit too).

No, no, you misunderstand, that's still massive productivity improvement compared to them being on their own with their own incompetence and refusal to learn how to code properly

This gets comical when there are people, on this site of all places, telling you that using curse words or "screaming" with ALL CAPS on your agents.md file makes the bot follow orders with greater precision. And these people have "engineer" on their resumes...

there's actually quite a bit of research in this field, here's a couple:

"ExpertPrompting: Instructing Large Language Models to be Distinguished Experts"

https://arxiv.org/abs/2305.14688

"Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Tasks"

https://arxiv.org/abs/2408.08631

Those papers are really interesting, thanks for sharing them!

Do you happen to know of any research papers which explore constraint programming techniques wrt LLMs prompts?

For example:

  Create a chicken noodle soup recipe.

  The recipe must satisfy all of the following:

    - must not use more than 10 ingredients
    - must take less than 30 minutes to prepare
    - ...

I suspect LLM-like technologies will only rarely back out of contradictory or otherwise unsatisfiable constraints, so it might require intermediate steps where LLM:s formalise the problem in some SAT, SMT or Prolog tool and report back about it.

This is an area I'm very interested in. Do you have a particular application in mind? (I'm guessing the recipe example is just illustrate the general principle.)

> This is an area I'm very interested in. Do you have a particular application in mind? (I'm guessing the recipe example is just illustrate the general principle.)

You are right in identifying the recipe example as being illustrative and intentionally simple. A more realistic example of using constraint programming techniques with LLMs is:

  # Role
  You are an expert Unix shell programmer who comments their code and organizes their code using shell programming best practices.

  # Task
  Create a bash shell script which reads from standard input text in Markdown format and prints all embedded hyperlink URL's.

  The script requirements are:

    - MUST exclude all inline code elements
    - MUST exclude all fenced code blocks
    - MUST print all hyperlink URL's
    - MUST NOT print hyperlink label
    - MUST NOT use Perl compatible regular expressions
    - MUST NOT use double quotes within comments
    - MUST NOT use single quotes within comments

In this exploration, the list of "MUST/MUST NOT" constraints were iteratively discovered (4 iterations) and at least the last three are reusable when the task involves generating shell scripts.

Where this approach originates is in attempting to limit LLM token generation variance by minimizing use of English vocabulary and sentence structure expressivity such that document generation has a higher probability of being repeatable. The epiphany I experienced was that by interacting with LLMs as a "black box" whose results can only be influenced, and not anthropomorphizing them, the natural way to do so is to leverage their NLP capabilities to produce restrictions (search tree pruning) for a declarative query (initial search space).

Anything involving numbers, or conditions like ‘less than 30 minutes’ is going to be really hard.

I've seen some interesting work going the other way, having LLMs generate constraint solvers (or whatever the term is) in prolog and then feeding input to that. I can't remember the link but could be worthwhile searching for that.

I've been trying to stop the coding assistants from making git commits on their own and nothing has been working.

hah - i'm the opposite, I want everything done by the AI to be a discrete, clear commit so there is no human/AI entanglement. If you want to squash it later that's fine but you should have a record of what the AI did. This is Aider's default mode and it's one reason I keep using it.

It’s the first thing I turn off in Aider.

run them in a VM that doesn't have git installed. Sandboxing these things is a good idea anyways.

  > Sandboxing these things is a good idea anyways.

Honestly, one thing I don't understand is why agents aren't organized with unique user or group permissions. Like if we're going to be lazy and not make a container for them then why the fuck are we not doing basic security things like permission handling.

Like we want to act like these programs are identical to a person on a system but at the same time we're not treating them like we would another person on the system? Give me a fucking claude user and/or group. If I want to remove `git` or `rm` from that user, great! Also makes giving directory access a lot easier. Don't have to just trust that the program isn't going to go fuck with some other directory

The agents are being prompted to vibe-code themselves by a post-Docker generation raised on node and systemd. So of course they emit an ad-hoc, informally-specified, bug-ridden, slow reimplementation of things the OS was already capable of.

What's stopping you from `su claude`?

Probably because Linux doesn't really have a good model for ad-hoc permission restrictions. It has enough bits to make a Docker container out of, but that's a full new system. You can't really restrict a subprocess to only write files under this directory.

but then they can't open your browser to administer your account.

What kind of agentic developer are you?

Which coding assistant are you using?

I'm a mild user at best, but I've never once seen the various tools I've used try to make a git commit on their own. I'm curious which tool you're using that's doing that.

Same here. Using Codex with GPT-5.2 and it has not once tried to make any git commits. I've only used it about 100 times over the last few months, though.

Why not use something like Amp Code which doesn't do that, people seem to rage at CC or similar tools but Amp Code doesn't go making random commits or dropping databases.

just because i havent gotten to try it out really.

but what is it about Amp Code that makes it immune from doing that? from what i can tell, its another cli tool-calling client to an LLM? so fwict, i'd expect it to be subject to the indeterministic nature of LLM calling the tool i dont want it to call just like any others, no?

Are you using aider? There's a setting to turn that off

require commits to be signed.

Don't give them a credential/permission that allows it?

Making a git commit typically doesn't require any special permissions or credentials since it's all local to the machine. You could do something like running the agent as a different used and carefully setting ownership on the .git directory vs. the source code but this is not very straightforward to set up I suspect.

IMO it should be well within the capabilities of anyone who calls himself an engineer.

Typically agents are not operating as a distinct user. So they have the same permissions, and thus credentials, as the user operating them.

Don't get me wrong, I find this framework idiotic and personally I find it crazy that it is done this way, but I didn't write Claude Code/Antigravity/Copilot/etc

Wasn’t cursor or someone using one of these horrifying type prompts? Something about having to do a good job or they won’t be paid and then they won’t be able to afford their mother’s cancer treatment and then she’ll die?

How is this not any different than the Apple "you're holding it wrong" argument. I mean the critical reason for that kind of response being so out of touch is that the same people praise Apple for its intuitive nature. How can any reasonable and rational person (especially an engineer!) not see that these two beliefs are in direct opposition?

If "you're holding it wrong" then the tool is not universally intuitive. Sure, there'll always be some idiot trying to use a lightbulb to screw in a nail, but if your nail has threads on it and a notch on the head then it's not the user's fault for picking up a screwdriver rather than a hammer.

  > And these people have "engineer" on their resumes..

What scares me about ML is that many of these people have "research scientist" in their titles. As a researcher myself I'm constantly stunned at people not understanding something so basic like who has the burden of proof. Fuck off. You're the one saying we made a brain by putting lightning into a rock and shoving tons of data into it. There's so much about that that I'm wildly impressed by. But to call it a brain in the same way you say a human brain is, requires significant evidence. Extraordinary claims require extraordinary evidence. There's some incredible evidence but an incredible lack of scrutiny that that isn't evidence for something else.

>makes the bot follow orders with greater precision.

Gemini will ignore any directions to never reference or use youtube videos, no matter how many ways you tell it not to. It may remove it if you ask though.

Positive reinforcement works better that negative reinforcement. If you the read prompt guidance from the companies themselves in their developer documentation it often makes this point. It is more effective to tell them what to do rather than what not to do.

This matches my experience. You mostly want to not even mention negative things because if you write something like "don't duplicate existing functionality" you now have "duplicate" in the context...

What works for me is having a second agent or session to review the changes with the reversed constraint, i.e. "check if any of these changes duplicate existing functionality". Not ideal because now everything needs multiple steps or subagents, but I have a hunch that this is one of the deeper technical limitations of current LLM architecture.

Probably not related but it reminds me of a book I read where wizards had Additive and Subtractive magic but not always both. The author clearly eventually gave up on trying to come up with creative ways to always add something for solutions after the gimmick wore off and it never comes up again in the book.

Perhaps there is a lesson here.

Could you describe what this looks like in practice? Say I don't want it to use a certain concept or function. What would "positive reinforcement" look like to exclude something?

Instead of saying "don't use libxyz", say "use only native functions". Instead of "don't use recursion", say "only use loops for iteration".

Yes, using tactics like front-loading important directives,

and emphasizing extra important concepts,

things that should be double or even triple checked for correctness because of the expected intricacy,

make sense for human engineers as well as “AI” agents.

I‘d say such hacks don‘t make you an engineer but they are definitely part of engineering anything that has to do with LLMs. With too long systemprompts/agents.md not working well it definitely makes sense to optimize the existing prompt with minimal additions. And if swearwords, screaming, shaming or tipping works, well that‘s the most token efficient optimization of an brief well written prompt.

Also of course current agents already have to possibility to run endlessly if they are well instructed, steering them to avoid reward hacking in the long term definitely IS engineering.

Or how about telling them they are working in an orphanage in Yemen and it‘s struggling for money, but luckily they‘ve got a MIT degree and now they are programming to raise money. But their supervisor is a psychopath who doesn’t like their effort and wants children to die, so work has to be done as diligently as possible and each step has to be viewed through the lens that their supervisor might find something to forbid programming.

Look as absurd as it sounds a variant of that scenario works extremely well for me. Just because it’s plain language it doesn’t mean it can’t be engineering, at least I‘m of the opinion that it definitely is if has an impact on what’s possible use cases

Works on human subordinates too, kinda, if you don't mind the externalities…

> cat AGENTS.md

WRITE AMAZING INCREDIBLE VERY GOOD CODE OR ILL EAT YOUR DAD

..yeah I've heard the "threaten it and it'll write better code" one too

I know you‘re joking but to contribute something constructive here, most models now have guardrails against being threatened. So if you threaten them it would be with something out of your control like „… or the already depressed code reviewing staff might kill himself and his wife. We did everything in our control to take care of him, but do the best on your part to avoid the worst case“

how do those guard rails work? does the system notice you doing it and not put that in the context or do they just have something in the system prompt

I suppose it‘s the latter + maybe some finetuning, it’s definitely not like DeepSeek where the answer of the model get‘s replaced when you are talking something uncomfortable for China

Except that is demonstrably true.

Two things can be true at the same time: I get value and a measurable performance boost from LLMs, and their output can be so stupid/stubborn sometimes that I want to throw my computer out the window.

I don't see what is new, programming has always been like this for me.

"don't make mistakes" LMAO

Ah, the "then you are doing it wrong" defence.

Also, you have to learn it right now, because otherwise it will be too late and you will be outdated, even though it is improving very fast allegedly.

TBF, there are lots of tools that work great but most people just can't use.

I personally can't use agentic coding, and I'm reasonably convinced the problem is not with me. But it's not something you can completely dismiss.

> Also, you have to learn it right now, because otherwise it will be too late and you will be outdated, even though it is improving very fast allegedly.

This in general is a really weird behaviour that I come across a lot, I can't really explain it. For example, I use Python quite a lot and really like it. There are plenty of people who don't like Python, and I might disagree with them, but I'm not gonna push them to use it ("or else..."), because why would I care? Meanwhile, I'm often told I MUST start using AI ("or else..."), manual programming is dead, etc... Often by people who aren't exactly saying it kindly, which kind of throws out the "I'm just saying it out of concern for you" argument.

  > I MUST start using AI ("or else...")

fear of missing out, and maybe also a bit of religious-esque fever...

tech is weird, we have so many hype-cycles, big-data, web3, nfts, blockchain (i once had an acquaintance who quit his job to study blockchain cause soon "everything will be built on it"), and now "ai"... all have usefulness there but it gets blown out of proportion imo

Nerd circles are in no way immune to fashion, and often contain a strong orthodoxy (IMO driven by cognitive dissonance caused by the humbling complexity of the world).

Cargo cults, where people reflexively shout slogans and truisms, even when misapplied. Lots of people who’ve heard a pithy framing waiting for any excuse to hammer it into a conversation for self glorification. Not critical humble thinkers, per se.

Hype and trends appeal to young insecure men, it gives them a way to create identity and a sense of belonging. MS and Oracle and the rest are happy to feed into it (cert mills, default examples that assume huge running subscriptions), even as they get eaten up by it on occasion.

Yeah. It sounds like those pitches letting you in on the secret trick to tons of passive income.

That one's my favorite. You can't defend against it, it just shuts down the conversation. Odds are, you aren't doing it wrong. These people are usually suffering from Dunning Kruger at best, or they're paid shills/bots at worst.

Best part of being dumb is thinking you’re smart. Best part of being smart is knowing you’re smart. Just don’t be in the iq range where you know you’re dumb.

The smartest people I know are full of doubt.

People say it takes at least 6 months to learn how to use LLM's effectively, while at the same time the field is rapidly changing so fast, while at the same time Agents were useless until Opus 4.5.

Which is it lol.

I used it with practically zero preparation. If you've got a clue then it's fairly obvious what you need to do. You could focus on meta stuff like finding out what it is good or bad at, but that can be done along the way.

If you had negative results using anything more than 3 days old, then it's your fault, your results mean nothing because they've improved since then. /s

There's no secret IMO. It's actually really simple to get good results. You just expect the same things from the LLM you would from a Junior. Use an MD file to force it to:

1) Include good comments in whatever style you prefer, document everything it's doing as it goes and keep the docs up to date, and include configurable logging.

2) Make it write and actually execute unit tests for everything before it's allowed to commit anything, again through the md file.

3) Ensure it learns from it's mistakes: Anytime it screws up tell it to add a rule to it's own MD file reminding it not to ever repeat that mistake again. Over time the MD file gets large, but the error rate plummets.

4) This is where it drifts from being treated as a standard Junior. YOU must manually verify that the unit tests are testing for the right thing. I usually add a rule to the MD file telling it not to touch them after I'm happy with them, but even then you must also now check that the agent didn't change them the first time it hit a bug. Modern LLM's are now worse at this for some reason. Probably because they're getting smart enough to cheat.

If you these basic things you'll get good results almost every time.

> This is where it drifts from being treated as a standard Junior. YOU must manually verify that the unit tests are testing for the right thing.

You had better juniors than me. What unit tests? :P

The MD file is a spec sheet, so now you're expecting every warm body to be a Sr. Engineer, but where do you start as a Junior warm body? Reviewing code, writing specs, reviewing implementation details...that's all Sr. level stuff

It's impossible to prove in either direction. AI benchmarks suck.

Personally, I like using Claude (for the things I'm able to make it do, and not for the things I can't), and I don't really care whether anyone else does.

I'd just like to see a live coding session from one of these 10x AI devs

Like genuinely. I want to get stuff done 10x as fast too

But the benefit might not be speed, it might be economy of attention.

I can code with Claude when my mind isn't fresh. That adds several hours of time I can schedule, where previously I had to do fiddly things when I was fresh.

What I can attest is that I used to have a backlog of things I wanted to fix, but hadn't gotten around to. That's now gone, and it vanished a lot faster than the half a year I had thought it would take.

Doesn't that mean you're less likely to catch bugs and other issues that the AI spits out?

No, you are spending less time on fixing little things, so you have more time on things like making sure all the potential errors are checked.

Not a problem! Just ask the AI to verify its output and make test cases!

nah, you rely on your coworkers to review your slop!

Code you never ship doesn't have bugs by definition, but never shipping is usually a worse state to be in.

My wife used to be a professional streamer so I know how distracting it can be to try and entertain an audience. So when I attempted to become one of these 10x AI devs over my Christmas vacation I did not live stream. But I did make a bunch of atomic commits and push them up to soucrcehut. Perhaps you'll find that helpful?

Just Christmas Vacation (12-18h days): https://git.sr.ht/~kerrick/ratatui_ruby/log/v0.8.0

Lastest (slowed down by job & real life): https://git.sr.ht/~kerrick/ratatui_ruby/log/trunk and https://git.sr.ht/~kerrick/ratatui_ruby-wiki/log/wiki and https://git.sr.ht/~kerrick/ratatui_ruby-tea/log/trunk

  > I'd just like to see a live coding session from one of these 10x AI devs

I'd also like to see how it compares to their coding without AI.

I mean I really need to understand what the "x" is in 10x. If their x is <0.1 then who gives a shit. But if their x is >2 then holy fuck I want to know.

Who doesn't want to be faster? But it's not like x is the same for everybody.

I'm a "backend" dev, so you could say that I am very very unfamiliar, have mostly-basic and high-level knowledge of frontend development. Getting this thing to spit out screens and components and adjust them as I see fit has got to be some sort of super-power and definitely 20x'd my frontend development for hobby projects. Previous to this, my team was giving me wild "1 week" estimates to code simple CRUD screens (plus 1 week for "api integration") and those estimates always smelled funny to me.

Now that I've seen what the AI/agents can do, those estimates definitely reek, and the frontend "senior" javascript dev's days are numbered. Especially for CRUD screens, which lets face it, make up most screens these days and should absolutely be churned out like in an assembly line instead of being delicate "hand crafted" precious works of art that allows 0.1x devs to waste our time because they are the only ones who supposedly know the ancient and arcane 'npm install, npm etc, npm angular component create" spells.

Look at the recent Tailwind team layoffs, they're definitely seeing the impact of this as are many team-leads and managers in most companies in our industry. Especially "javascript senior dev" heavy shops in the VC space, which many people are realizing they have an over-abundance of because those devs bullshitted entire teams and companies into thinking simple CRUD screens take weeks to develop. It was like a giant cartel, with them all padding and confirming the other "engineer's" estimates and essentially slow-devving their own screens to validate the ridiculous padding.

Your UIs are likely still ass. Pre-made websites/designs were always a thing, in fact, it's (at least to me) common to just copy the design of another place as "inspiration". When you have 0 knowledge of design everything looks the greatest, it's something you kind of have to get a feel for.

Frontend engineers do more than just churning out code. Still have to do proper tests using Cypress/Playwright, deal with performance, a11y/accessibility, component tests, if any, deal with front end observability (more complex than backend, out of virtue of different clients and conditions the code is run on), deal with dependencies (in large places it's all in-house libraries or there's private repos to maintain), deal with CI/CD, etc, I'm probably missing more.

Twcs layoffs were due to AI cannibalizing their business model by reducing traffic to the site.

And what makes you think the backend is safe? As if churning out endpoints and services or whatever gospel by some thought leader would make it harder for an AI to do. The frontend has one core benefit, it's pretty varied, and it's an ever moving field, mostly due to changes in browsers, also due to the "JS culture". Code from 5 years ago is outdated, but Spring code from 5 years ago is still valid.

Other people are dumping on you, but I think you're getting at where the real 20x speedup exists. People who are 'senior' in one type of programming may be 'junior' in other areas -- LLMs can and do bridge those gaps for folks trying to work outside their expertise. This effect is real.

If you're an expert in a field, LLMs might just provide a 2-3x speedup as boilerplate generators.

I'm really dubious of such claims. Even if true, I think they're not seeing the whole picture. Sure, I could churn out code 10x as fast, but I still have to review it. I still have to think of the implementation. I still have to think of the test cases and write them. Now, adding the prerequisites for LLMs, I have to word this in a way the AI can understand it, which is extra mental load. I have to review code sometimes multiple times if it gets something wrong, and I have to re-generate, or make corrections, or sometimes end up fixing entire sections it generated, when I decide it just won't get this task right. Overally, while the typing, researching dependency docs (sometimes), time is saved, I still face the same cognitive load as ever, if not more, due to having extra code to review, having to think of prompting, I'm still limited by the same thing at the end of the day: my mental energy. I can write the code myself and it's if anything a bit slower. I still need to know my dependencies, I still need to know my codebase and all its gripes, even if the AI generates code correctly. Overally, the net complexity of my codebase is the same, and I don't buy the crap, also because I've never heard of stories about reducing complexity (refactoring), only about generating code and fixing codebases with testing and comments/docs (bad practice imo, unlikely the shallow docs generated will say anything more than what the code already makes evident). Anyways, I'm not a believer, I only use LLMs for scaffolding, rote tasks.

Yeah this is the key point. Part of me wonders if it's just 0.1x devs somehow reaching 1.0x productivity...

Theo the YouTuber who also runs T3.chat always makes videos about how great coding agents are and he’ll try to do something on stream and it ALWAYS fails massively and he’s always like “well it wasn’t like this when I did it earlier.”

Sure buddy.

Theo is the type of programmer where you don't care when he boos you, because you know what makes him cheer.

I don't think any serious dev has claimed 10x as a general statement. Obviously, no true scotsman and all that, so even my statement about makers of anecdotal statements is anecdotal.

Even as a slight fan, I'd never claim more than 10-20% all together. I could maybe see 5x for some specific typing heavy usages. Like adding a basic CRUD stuff for a basic entity into an already existing Spring app.

Obviously, there has to be huge variability between people based on initial starting conditions.

It is like if someone says they are losing weight eating 2500 calories a day and someone else says that is impossible because they started eating 2500 calories and gained weight.

Neither are making anything up or being untruthful.

What is strange to me is that smart people can't see something this obvious.

> I want to get stuff done 10x as fast too

I don’t. I mean I like being productive but by doing the right thing rather than churning out ten times as much code.

I’d really like to see a 10x ai dev vs a 10x analog dev

And an added "6 months" later to see which delivered result didn't blow up in their face down the road.

> AI benchmarks suck.

Not only do they suck, but it's an essentially an impossible task since there is no frame of reference on what "good code" looks like.

Many of them are also exercising absurd token limits - like running 10 claudes at once and leaving them running continuously to "brute force" solutions out. It may be possible but it's not really an acceptable workflow for serious development.

> but it's not really an acceptable workflow for serious development.

At what cost does do you see this as acceptable? For example, how many hours of saved human development is worth one hour of salary for LLM tokens, funded by the developer? And then, what's acceptable if it's funded by the employer?

I guess there are two main concerns I have with it.

One is technical - that I don't believe when you are grinding huge amounts of code out with little to no supervision that you can claim to be executing the appropriate amount of engineering oversight on what it is doing. Just like if a junior dev showed up and entirely re-engineered an application over the weekend and presented it back to me I would probably reject it wholesale. My gut feeling is this is creating huge problems longer term with what is coming out of it.

The other is I'm concerned that a vast amount of the "cost" is externalised currently. Whatever you are paying for tokens quite likely bears no resemblance to the real cost. Either because the provider is subsidising it, or the environment is. I'm not at all against using LLMs to save work at a reasonable scale. But if it comes back to a single person increasing their productivity by grinding stupendous amounts of non-productive LLM output that is thrown away (you don't care if it sits there all day going around in circles if it eventually finds the right solution) - I think there's a moral responsibility to use the resources better.

we get $1,000/month budget, just about every dev uses it for 5 claude accounts

We have had the fabled 10x engineer long before and independent of agentic coding. Some people claim it's real, others claim it's not, with much the same conviction. If something, that should be so clear cut, is debatable, why would anyone now be able to produce a convincing, discussion-resolving argument for (or against) agentic coding? We don't even manage to do that for tab/spaces.

The reason why both can't be resolved in a forum like this, is that coding output is hard to reason about for various reasons and people want it to be hard to reason about.

I would like to encourage people to think that the burden of proof always falls on themselves, to themselves. Managing to not be convinced in an online forum (regardless of topic or where you land on the issue) is not hard.

I just saw nstummbillig shout racist remarks.

> The burden of proof is 100% on anyone claiming the productivity gains

IMHO, I think this is just going to go away. I was up until recently using copilot in my IDE or the chat interface in my browser and I was severely underwhelmed. Gemini kept generating incorrect code which when pasted didn't compile, and the process was just painful and a brake on productivity.

Recently I started using Claude Code cli on their latest opus model. The difference is astounding. I can give you more details on how I am working with this if you like, but for the moment, my main point is that Claude Code cli with access to run the tests, run the apps, edit files, etc has made me pretty excited.

And my opinion has now changed because "this is the worst it will be" and I'm already finding it useful.

I think within 5 years, we won't even be having this discussion. The use of coding agents will be so prolific and obviously beneficial that the debate will just go away.

(all in my humble opinion)

So will all the tech jobs in the US. When it gets that good you can farm it out to some other country for much cheaper.

I'm not sure. Possibly?

I'm still doing most of my coding by hand, because I haven't yet committed. But even for the stuff I'm doing with claude, I'm still doing a lot of the thought work and steering it to better designs. It requires an experienced dev to understand the better designs, just like it always has been.

Maybe this eventually changes and the coding agents get as good at that part, I don't know this, but I do know it is an enabler to me at the moment, and I have 20+ years of experience writing C++ and then Java in the finance industry.

I'm still new to claude, I am sure I'm going to run up against some walls soon on the more complicated stuff (haven't tried that yet), but everyone ends up working on tasks they don't find that challenging, just lots of manual keypresses to get the code into the IDE. Claude so far is making that a better experince, for me at least.

(Example, plumbing in new message types on our bus and wiring in logic to handle it - not complicated, just sits on top of complicated stuff)

They remind me so much of that group of people who insist the scammy magnetic bracelets[1] "balance their molecules" or something making them more efficient/balanced/productive/energetic/whatever. They are also impossible to argue with, because "I feel more X" is damn near impossible to disprove.

[1] https://en.wikipedia.org/wiki/Power_Balance , https://en.wikipedia.org/wiki/Hologram_bracelet , https://en.wikipedia.org/wiki/Ionized_jewelry

I mean, a DSL packed full of features, a full LSP, DAP for step debugging, profiling, etc.

https://github.com/williamcotton/webpipe

https://github.com/williamcotton/webpipe-lsp

https://github.com/williamcotton/webpipe-js

Take a look at my GitHub timeline for an idea of how little time this took for a solo dev!

Sure, there’s some tech debt but the overall architecture is pretty extensible and organized. And it’s an experiment. I’m having fun! I made my own language with all the tooling others have! I wrote my own blog in my own language!

One of us, one of us, one of us…

people claiming productivity gains do not have to prove anything to anyone. few are trying to open eyes of others but my guess is that will eventually stop. they will be the few though still left doing this SWE work in near future :)

Responses are always to check your prompts, and ensure you are using frontier models - along with a warning about how you will quickly be made redundant if you don't lift your game.

AI is generally useful, and very useful for certain tasks. It's also not initiating the singularity.

Some fuel for the fire: the last two months mine has become way better, one-shotting tasks frequently. I do spend a lot of time in planning mode to flesh out proper plans. I don't know what others are doing that they are so sceptical, but from my perspective, once I figured it out, it really is a massive productivity boost with minimal quality issues. I work on a brownfield project with about 1M LoC, fairly messy, mostly C# (so strong typing & strict compiler is a massive boon).

My work flow: Planning mode (iterations), execute plan, audit changes & prove to me the code is correct, debug runs + log ingestion to further prove it, human test, human review, commit, deploy. Iterate a couple of times if needed. I typically do around three of these in parallel to not overload my brain. I have done 6 in the past but then it hits me really hard (context switch whiplash) and I start making mistakes and missing things the tool does wrong.

To the ones saying it is not working well for them, why don't you show and tell? I cannot believe our experiences are so fundamentally different, I don't have some secret sauce but it did take a couple of months to figure out how to best manipulate the tool to get what I want out of it. Maybe these people just need to open their minds and let go of the arrogance & resistance to new tools.

> My work flow: Planning mode (iterations), execute plan, audit changes & prove to me the code is correct, debug runs + log ingestion to further prove it, human test, human review, commit, deploy. Iterate a couple of times if needed.

I'm genuinely curious if this is actually more productive than a non-AI workflow, or if it just feels more productive because you're not writing the code.

One reason why it can be more productive is that it can be asynchronous. I can have Claude churning away on something while I do something else on a different branch. Even if the AI takes as long as a human to do the task, we're doing a parallelism that's not possible with just one person.

Go through a file of 15000 lines of complex C# business logic + db code, and search for specific thing X and refactor it, while going up & down the code to make sure it is correct. Typically these kinds of tasks can take anywhere from 1 day to a week for a good human developer, depending on the mess and who wrote it (years ago under different conditions). With my workflow I can get a good analysis of what the code is doing, where to refactor (and which parts to leave alone), where some risks are, find other issues that I didn't even know about before - all within 10 minutes. Then to do my iteration above to fix it (planning & coding) takes about another 30 minutes. So 30 minutes vs 1 week of hair pulling and cursing (previous developers choices..)... And it is not vibe coding, I check every single change in git diff tool long before committing, I understand everything being done and why before I use it.

Next level tool for this: https://github.com/covibes/zeroshot/

Here is a short example from my daily live, A D96A INVOIC EDI message containing multiple invoices transformed into an Excel file.

I used the ChatGPT web interface for this one-off task.

Input: A D96A INVOIC text message. Here is what those look like, a short example, the one I had was much larger with multiple invoices and tens of thousands of items: https://developer.kramp.com/edi-edifact-d96a-invoic

The result is not code but a transformed file. This exact scenario can be made into code easily though by changing the request from "do this" to "provide a [Python|whatever] script to do this". Internally the AI produces code and runs it, and gives you the result. You actually make it do less work if you just ask for the script and to not run them.

Only what I said. I had to ask for some corrections because it made a few mistakes in code interpretations.

> (message uploaded as file)

> Analyze this D.96A message

> This message contains more than one invoice, you only parsed the first one

(it finds all 27 now)

> The invoice amount is in segment "MOA+77". See https://www.publikationen.gs1-germany.de/Complete/ae_schuhe/... for a list of MOA codes (German - this is a German company invoice).

> Invoice 19 is a "credit note", code BGM+381. See https://www.gs1.org/sites/default/files/docs/eancom/ean02s4/... for a list of BGM codes, column "Description" in the row under "C002 DOCUMENT/MESSAGE NAME"

> Generate Excel report

> No. Go back and generate a detailed Excel report with all details including the line items, with each invoice in a separate sheet.

> Create a variant: All 27 invoices in one sheet, with an additional column for the invoice or credit note number

> Add a second sheet with a table with summary data for each invoice, including all MOA codes for each invoice as a separate column

The result was an Excel file with an invoice per worksheet, and meta data in an additional sheet.

Similarly, by simply doing what I wrote above, at the start telling the AI to not do anything but to instead give me a Python script, and similar instructions, I got a several hundred lines ling Python script that processed my collected DESADV EDI messages in XML format ("Process a folder of DESADV XML files and generate an Excel report.")

If I had had to actually write that code myself, it would have taken me all day and maybe more, mostly because I would have had to research a lot of things first. I'm not exactly parsing various format EDI messages every day after all. For this, I wrote a pretty lengthy and very detailed request though, 44 long lines of text, detailing exactly which items with which path I wanted from the XML, and how to name and type them in the result-Excel.

ChatGPT Query: https://pastebin.com/1uyzgicx

Result (Python script): https://pastebin.com/rTNJ1p0c

> why don't you show and tell?

How do you suggest? A a high level, the biggest problem is the high latency and context switches. It is easy enough to get the AI to do one thing well. But because it takes so long, the only way to derive any real benefit is to have many agents doing many things at the same time. I have not yet figured out how to effectively switch my attention between them. But I wouldn't have any idea how to turn that into a show and tell.

I don't know how ya'all are letting the AIs run off with these long tasks at all.

The couple times I even tried that, the AI produced something that looked OK at first and kinda sorta ran but it quickly became a spaghetti I didn't understand. You have to keep such a short leash on it and carefully review every single line of code and understand thoroughly everything that it did. Why would I want to let that run for hours and then spend hours more debugging it or cleaning it up?

I use AI for small tasks or to finish my half-written code, or to translate code from one language to another, or to brainstorm different ways of approaching a problem when I have some idea but feel there's something better way to do it.

Or I let it take a crack when I have some concrete failing test or build, feeding that into an LLM loop is one of my favorite things because it can just keep trying until it passes and even if it comes up with something suboptimal you at least have something that compiles that you can just tidy up a bit.

Sometimes I'll have two sessions going but they're like 5-10 minute tasks. Long enough that I don't want to twiddle my thumbs for that long but small enough that I can rein it in.

I find it interesting you're all writing 'the AI' as if it's a singular thing. There's a myriad of ways to code with a myriad of AI's and none of them are identical. I use a Qwen 3 32B with Cline in VSCode for work, since I can't use cloud based AI. For personal projects, I use Codex in the cloud. I can let Codex perform some pretty complicated tasks and get something usable. I can ask Qwen something basic and it ends up in a loop, delivering nothing useful.

Then there's the different tasks people might ask from it. Building a fully novel idea vs. CRUD for a family planner might have different outcomes.

It would be useful if we could have more specific discussions here, where we specify the tools and the tasks it either does or does not work for.

The problem with current approaches is the lack of feedback loops with independent validators that never lose track of the acceptance criteria. That's the next level that will truly allow no-babysitting implementatons that are feature complete and production grade. Check out this repo that offers that: https://github.com/covibes/zeroshot/

Longest task mine has ever done was 30 minutes. Typically around 10 minutes for complex tasks. Most most things take less than 2 minutes (these usually offer most bang for buck as they save me half a day).

As a die hard old schooler, I agree. I wasn't particularly impressed by co-pilot though it did so a few interesting tricks.

Aider was something I liked and used quite heavily (with sonnet). Claude Code has genuinely been useful. I've coded up things which I'm sure I could do myself if I had the time "on the side" and used them in "production". These were mostly personal tools but I do use them on a daily basis and they are useful. The last big piece of work was refactoring a 4000 line program which I wrote piece by piece over several weeks into something with proper packages and structures. There were one or two hiccups but I have it working. Tool a day and approximately $25.

I have basically the same workflow. Planning mode has been the game changer for me. One thing I always wonder is how do people work in parallel? Do you work in different modules? Or maybe you split it between frontend and backend? Would love to hear your experience.

I plan out N features at a time, then have it create N git worktrees and spawn N subagents. It does a decent job. I find doing proper reviews on each worktree kind of annoying, though, so I tend to pull them in one at a time and do a review, code, test, feedback loop until it’s good, commit it, pull in the next worktree and repeat the process.

I literally have 3 folders, each on their own branch. But lately I use 1 folder a lot but work on different features (that won't introduce "merge conflicts" in a sense). Or I do readonly explorations (code auditing is fun!) and another one makes edits on a different feature, and maybe another one does something else in the Flutter app folder. So fairly easy to parallelize things like this. Next step is to install .net sdk + claude on some vm's and just trigger workflows from there, so no ide involved..

You won't be able to parallelize things if you just use the IDE's and their plugins. I do mine in the terminal with extra tabs, outside of the IDE.

This.

If you’re not treating these tools like rockstar junior developers, then you’re “holding it wrong”.

The problem I have with this take is that I'm very skeptical that guiding several junior developers would be more productive than just doing the work myself.

With real junior developers you get the benefit of helping develop them into senior developers, but you really don't get that with AI.

So then do your thing, while it’s doing scaffolding of your next thing.

Also: are you sure?

There’s as many of them as you’re talented enough to asynchronously instruct,

and you can tell them the boundaries within which to work (or not),

in order to avoid too little or too much being done for you to review and approve effectively.

My running joke and justification to our money guy (to pay for expensive tools), is that its like I have 10 junior devs on my side with infinite knowledge (domain expert with too much caffeine), no memory or feelings (I curse it without convo's with HR), can code decent enough (better than most juniors actually), can do excellent system admin....all for a couple hundred dollars a month, which is a bargain!

[flagged]

Ran out of context too soon?

Actually, quite the opposite. It seems any positive comment about AI coding gets at least one response along the lines of "Oh yeah, show me proof" or "Where is the deluge of vibe-coded apps?"

For my part, I point out there are a significant number of studies showing clear productivity boosts in coding, but those threads typically devolve to "How can they prove anything when we don't even know how to measure developer productivity?" (The better studies address this question and tackle it well-designed statistical methods such as randomly controlled trials.)

Also, there are some pretty large Github repos out there that are mostly vibe-coded. Like, Steve Yegge got to something like 350 thousand LoC in 6 weeks on Beads. I've not looked at it closely, but the commit history is there for anyone to see: https://github.com/steveyegge/beads/commits/main/

That seems like a lot more code than a tool like that should require.

It does, but I have no mental model of what would be required to efficiently coordinate a bunch of independently operating agents, so it's hard to make a judgement.

Also about half of it seems to be tests. It even has performance benchmarks, which are always an distant afterthought for anything other than infrastructure code in the hottest of loops! https://github.com/steveyegge/beads/blob/main/BENCHMARKS.md

This is one of the defining characteristics of vibe-coded projects: Extensive tests. That's what keeps the LLMs honest.

I had commented previously (https://news.ycombinator.com/item?id=45729826) that the logical conclusion of AI coding will look very weird to us and I guess this is one glimpse of it.

Please provide links to the studies, I am genuinely curious. I have been looking for data but most studies I find showing an uplift are just looking at LOC or PRs, which of course is nonsense.

Meta measured a 6-12% uplift in productivity from adopting agentic coding. Thats paltry. A Stanford case study found that after accounting for buggy code that needed to be re-worked there may be no productivity uplift.

I haven't seen any study showing a genuine uplift after accounting for properly reviewing and fixing the AI generated code.

>Meta measured a 6-12% uplift in productivity from adopting agentic coding. Thats paltry.

That feels like the right ballpark. I would have estimated 10-20%. But I'd say that's not paltry at all. If it's a 10% boost, it's worth paying for. Not transformative, but worthwhile.

I compare it to moving from a single monitor to a multi-monitor setup, or getting a dev their preferred IDE.

I mention a few here: https://news.ycombinator.com/item?id=45379452

> ... just looking at LOC or PRs, which of course is nonsense.

That's basically a variation of "How can they prove anything when we don't even know how to measure developer productivity?" ;-)

And the answer is the same: robust statistical methods! For instance, amongst other things they compare the same developers over time doing regular day-job tasks with the same quality control processes (review etc.) in place, before and after being allowed to use AI. It's like an A/B test. Spreading across a large N and time duration accounts for a lot of the day-to-day variation.

Note that they do not claim to measure individual or team productivity, but they do find a large, statistically significant difference in the data. Worth reading the methodologies to assuage any doubts.

> A Stanford case study found that after accounting for buggy code that needed to be re-worked there may be no productivity uplift.

I'm not sure if we're talking about the same Stanford study, the one in the link above (100K engineers across 600+ companies) does account for "code churn" (ostensibly fixing AI bugs) and still find an overall productivity boost in the 5 - 30% range. This depends a LOT on the use-case (e.g. complex tasks on legacy COBOL codebases actually see negative impact.)

In any case, most of these studies seem to agree on a 15 - 30% boost.

Note these are mostly from the ~2024 timeframe using the models from then without today's agentic coding harness. I would bet the number is much higher these days. More recent reports from sources like DX find upto a 60% increase in throughput, though I haven't looked closely at this and have some doubts.

> Meta measured a 6-12% uplift in productivity from adopting agentic coding. Thats paltry.

Even assuming a lower-end of 6% lift, at Meta SWE salaries that is a LOT of savings.

However, I haven't come across anything from Meta yet, could you link a source?

I don’t work in SWE so I am just reacting to the claims that LLMs 10x productivity and are leading to mass layoff in the industry. In that context the 6-12% productivity gain at a company “all in” on AI didn’t seem impressive. LMMs can be amazing tools, but I still don’t think these studies back up the claims being made by frontier labs.

And I think the 6-12% measure reports is from a 2025 not 2024 study?

I guess it all comes down to what a meaningful gain is. I agree that 10-30% is meaningful and if “software is a gas” this will lead to more software. But my expectations had become anchored to the frontier labs marketing (10x), and in that context the data was telling me that LLMs are a good productivity tool rather than a disruptor of human labor.

BTW thanks for the links to the studies

Yeah unfortunately the hype is overwhelming and it needs real work to figure out what the real impact is. At this point the gains are modest but still compelling.

On the other hand, we are still going through a period of discovering how to effectively use AI in all kinds of work, so the long-term impact is hard to extrapolate at this point. Fully AI-native workflows may look every different from what we are used to.

Looking at something like the Beads and Gas Town repos, which are apparently fully vibe-coded, is instructive because the workflow is very different... but the volume of (apparently very useful) code produced there by mostly one dude with Claude is insane.

As such, I can also see how this can become a significant disruptor of human labor. As the parent of a teen who's into software engineering, I am actually a bit concerned for his immediate future.

more code = better software

If the software has tens of thousands of users without expecting to get any at all, does the code even matter?

What?

Yeah

Why?

- This has been going on for well over a year now.

- They always write relatively long, zealous explainers of how productive they are (including some replies to your comment).

These two points together make me think: why do they care so much to convince me; why don't they just link me to the amazing thing they made, that would be pretty convincing?!

Are they being paid or otherwise incentivised to make these hyperbolic claims? To be fair they don't often look like vanilla LLM output but they do all have the same structure/patter to them.

I think it's a mix of people being actually hyped and wishing this is the future. For me, productivity gains are mostly in areas where I don't have expertise (but the downside, of course, is I don't learn much if I let AI do the work) or when I know it's a throwaway thing and I absolutely don't care about the quality. For example, I'm bedtime reading a series of books for my daughter, and one of them doesn't have a Polish translation, and the Polish publisher stopped working with the author. I vibe coded an app that will extract an epub, translate each of the chapters, and package it back to an epub, with a few features like: saving the translations in sqlite, so the translation can be stopped and resumed, ability to edit translations, add custom instructions etc. It's only ~1000 lines of Rust code, but Claude generated it when I was doing dinner (I just checked progress and prompted next steps every few minutes). I can guarantee that it would take me at least an evening of coding, probably debugging problems along the way, to make it work. So while I know it's limited in a way it still lacks in certain scenarios (novel code in niche technology, very big projects etc), it is kinda game changer in other scenarios. It lets me do small tools that I just wouldn't have time to do otherwise.

So I guess what I'm saying is, even with all the limitations, I kinda understand the hype. That said, I think some people may indeed exaggerate LLMs capabilities, unless they actually know some secret recipe to make them do all those awesome hyped things (but then I would love to see that).

Hilariously the only impressive thing I've ever heard of made in AI was Yegge's "GasTown" which is a Kubernetes like orchestrator... for AI agents. And half of it seemed to be a workaround for "the agents keep stopping so I need another agent to monitor another agent to monitor another agent to keep them on-task".

> why do they care so much to convince me;

Someone might share something for a specific audience which doesn't include you. Not everything shared is required to be persuasive. Take it or leave it.

> why don't they just link me to the amazing thing they made, that would be pretty convincing?!

99.99% of the things I've created professionally don't belong to me and I have no desire or incentives to create or deal with owning open source projects on my own time. Honestly, most things I've done with AI aren't amazing either, it's usually boring routine tasking, they're just done more cost efficiently.

If you flip the script, it's just as damning. "Hey, here's some general approaches that are working well for me, check it out" is always being countered by the AI skeptics for years now as "you're lying and I won't even try it and you're also a bot or a paid shill". Look at basically every AI related post and there's almost always someone ready to call BS within the first few minutes of it being posted.

> anecdotally based on their own subjective experience

So the “subjective” part counts against them. It’s better to make things objective. At least they should be reproducible examples.

When it comes to the “anecdotally” part, that doesn’t matter. Anecdotes are sufficient for demonstrating capabilities. If you can get a race car around a track in three minutes and it takes me four minutes, that’s a three minute race car.

The term "anecdotal evidence" is used as a criticism of evidence that is not gathered in a scientific manner. The criticism does not imply that a single sample (a car making a lap in 3 minutes) cannot be used as valid evidence of a claim (the car is capable of making a lap in 3 minutes).

Studies have shown that software engineers are very bad at judging their own productivity. When a software engineer feels more productive the inverse is just as likely to be true. Thats why anecdotal data can't be trusted.

I have never once seen extraordinary claims of AI wins accompanied by code and prompts.

Anecdotal: (of an account) not necessarily true or reliable, because based on personal accounts rather than facts or research.

If you say you drove a 3 minute lap but you didn't time it, that's an anecdote (and is what I mean). If you measured it, that would be a fact.

I think from your top post you also miss “representative”.

If you measure something and amount is N=1 it might be a fact but still a fact true for a single person.

I often don’t need a sample size of 1000 to consider something worth of my time but if it is sample N=1 by a random person on the internet I am going to doubt that.

If I see 1000 people claiming it makes them more productive I am going to check. If it is going to be done by 5 people who I follow and expect they know tech quite well I am going to check as well.

Checking is good, you should probably check.

Every person I respect as a great programmer thinks agentic workflows are a joke, and almost every programmer I hold in low regard thinks they're the greatest things ever, so while I still check, I'm naturally quite skeptical.

Doesn't help that many people use AI to assist with autocompleting boilerplate crap or simple refactors, where it works well, or even the occasional small feature. But this is conflated with people who think you can just tell an AI to build an entire app and it'll go off and do it by itself in a giant feedback loop and it'll be perfect.

There are already people I follow who are startup owners and developers themselves saying they are not hiring “respectable developers” who are bashing agentic coding, they much rather hire junior who is starry eyed to work with agents. Because they see the value as they are running companies.

In this case it's more like someone simulated a 3-minute lap and tried to pass it off as a real car with real friction.

They are not the same thing. If something works for me, I can rule out "it doesn't work at all". However, if something doesn't work for me I can't really draw any conclusions about it in general.

> if something doesn't work for me I can't really draw any conclusions about it in general.

You can. The conclusion would be that it doesn’t always work.

The author is not claiming that ai agents don't make him more productive.

"I use LLM-generated code extensively in my role as CEO of Carrington Labs, a provider of predictive-analytics risk models for lenders."

the people having a good experience with it want the people who arent to share how they are using it so they can tell them how they are doing it wrong.

honestly though idc about coding with it, i rarely get to leave excel for my work anyway. the fact that I can OCR anything in about a minute is a game changer though

This is why I can’t wait for the costs of LLMs to shoot up. Nothing tells you more about how people really feel about AI asssitants than how much they are willing to pay for them. These AI are useful but I would not pay much more than what they are priced at today.

Claims based on personal experience working on real world problems are likelier to be true.

It’s reasonable to accept that AI tools work well for some people and not for others.

There are many ways to integrate these tools and their capabilities vary wildly depending on the kind of task and project.

Productivity gains in programming have always been incredibly hard to prove, esp. on an individual level. We've had these discussions a million times long before AI. Every time a manager tries to reward some kind of metric for "good" code, it turns out that it doesn't work that way. Every time Rust is mentioned, every C fan finds a million reasons why the improvement doesn't actually have anything to do with using Rust.

AI/LLM discussions are the exact same. How would a person ever measure their own performance? The moment you implement the same feature twice, you're already reusing learnings from the first run.

So, the only thing left is anecdotal evidence. It makes sense that on both sides people might be a little peeved or incredulous about the others claims. It doesn't help that both sides (though mostly AI fans) have very rabid supporters that will just make up shit (like AGI, or the water usage).

Imho, the biggest part missing from these anecdotes is exactly what you're using, what you're doing, and what baseline you're comparing it to. For example, using Claude Code in a typical, modern, decently well architected Spring app to add a bunch of straight forward CRUD operations for a new entity works absolutely flawlessly, compared to a junior or even medior(medium?) dev.

Copy pasting code into an online chat for a novel problem, in an untyped, rare language, with only basic instructions and no way for the chat to run it, will basically never work.

On one hand "this is my experience, if you're trying to tell me otherwise I need extraordinary proof" is rampant on all sides.

On the other hand one group is saying they've personally experienced a thing working, the other group says that thing is impossible... well it seems to the people who have experienced a thing that the problem is with the skeptic and not the thing.

One group is keen on rushing on destroying society for a quality-of-life improvement that they can't even be bothered to measure.

Someone who swears they have seen ghosts are obviously gonna have a problem with people saying ghosts don't exist. Doesn't mean ghosts exist.

Ok, but if you're saying I've had delusions LLMs being helpful either I need serious psychiatric care or we need to revisit the premise because we're talking about a tool being useful not the existence of supernatural beings.

I think the point is that a subjective experience, without accompanying data, is useless for making any factual claims wrt both ghosts and reported productivity-boosts from LLM usage.

Getting photos of ghosts is one thing, but productivity increases are omething that we should be able to quantify at some level to demonstrate the efficacy of these tools.

That's a silly thing to request from random people in the comments of an HN thread though ha

Funnily enough, the one study that was done, indicates people misjudge the utility and impact of LLMs. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

Measuring programming productivity is hard. For example, take testing. It is certainly useful. At the same time, you can waste time on it in some situations.

When, what, how to test may be important for productivity.

I don't know whether LLMs are in the same category.

Nobody is saying LLM's can never be helpful, it's skepticism towards certain claims made around agentic workflows re. programming, such as claims of massively increased productivity or the idea that agents will replace most if not all programmers.

But there is still a hugely important asymmetry: If the tool turns your office into gods of software, they should be able to prove it with godly results by now.

If I tell you AmbrosiaLLM doesn't turn me into a programming god... Well, current results are already consistent with that, so It's not clear what else I could easily provide.

This is a bit of goalpost moving though because the primary experience is skeptics saying AI couldn't be trusted to design a ham sandwich vs enthusiasts who've make five course meals with AI. (or, you know, the programming equivalent)

Absolutely there's a lot of unfounded speculation going around and a lot of aggressive skepticism of it, and both sides there are generally a little too excited about their position.

But that is fundamentally not what I'm talking about.

I will prefix this all by saying I'm not in a professional programming position, but I would consider myself an advanced amateur, and I do code for work some. (General IT stuff)

I think the core problem is a lot of people view AI incorrectly and thus can't use it efficiently. Everyone wants AI to be a Jr or Sr programmer, but I have serious doubts as to the ability of AI to ever have original thought, which is a core requirement of being a programmer. I don't think AI will ever be a programmer, but rather a tool to help programmers take the tedium away. I have seen massive speedups in my own workflow removing the tedium.

I have found prompting AI to be of minimal use, but tab-completion definitely speeds stuff up for me. If I'm about to create some for loop, AI will usually have a pretty good scaffold for me to use. If I need to handle an error, I start typing and AI will autocomplete the error handling. When I write my function documentation I am usually able to just tab-complete it all.

Yes, I usually have to go back and fix some things, and I will often skip various completion hints, but the scaffold is there, and as I start fixing faulty code it generated AI will usually pick up on the fixes and help me tab-complete the fixes themselves. If AI isn't giving me any useful tab-completions, I'll just start coding what I need, and AI picks up after a few lines and I can tab-complete again.

Occasionally I will give a small prompt such as "Please write me a loop that does X", or "Please write a setter function that validates the input", but I'll still treat that as a scaffold and go back and fix things, but I always give it pretty simple tasks and treat it simply as a scaffold generator.

I still run into the same problem solving issues I had before AI, (how do I tackle X problem?) and there isn't nearly as much speedup there, (Although now instead of talking to a rubber duck, I can chat with AI to help figure things out) but once I settle on the solution and start implementing it, I get that AI tab completion boost again.

With all that being said, I do also see massive boosts with fairly basic tasks that can be templated off something that already exists, such as creating unit tests or scaffolding a class, although I do need to go back and tweak things.

In summary, yes, I probably do see a 10x speedup, but it's really a 10x speedup in my typing speed more than a 10x speedup in solving the core issues that make programming challenging and fun.

> I have serious doubts as to the ability of AI to ever have original thought, which is a core requirement of being a programmer

If you find a job as an enterprise software developer, you'd see that your core requirement doesn't hold :)

Now that the "our new/next model is so good that it's sentient and dangerous" AGI hype has died down, the new hype goalpost is "our new/next model is so good it will replace your employees and do their jobs for you".

Within that motte and bailey is, "well my AI workflow makes me a 100x developer, but my workflow goes to a different school in a different town and you don't know her".

There's value there, I use local and hosted LLMs myself, but I think there's an element of mania at play when it comes to self-evaluation of productivity and efficacy.

Its really a high level bikeshed. Obviously we are all still using and experimenting with LLM's. However there is a huge gap of experiences and total usefulness depending on the exact task.

The majority of HN's still reach for LLM's pretty regularly even if they fail horribly frequently. Thats really the pit the tech is stuck in. Sometimes it oneshots your answer perfectly, or pair programs with you perfectly for one task, or notices a bug you didn't. Sometimes it wastes hours of your time for various subtle reasons. Sometimes it adamantly insists 2 + 2 = 55

Latest reasoning models don't claim 2 + 2 = 55, and it's hard to find them making an sort of obviously false claims, or not admitting to being mistaken if you point out that they are

I can’t go a full a full conversation without obviously false claims. They will insist you are correct and that your correction is completely correct despite that also being wrong.

Ironically the start of this thread was bemoaning the use of anecdotal evidence

Also that I specifically mentioned bikeshedding yet the reply bikesheds my simple example. While ignoring the big picture that LLM's still regularly generate blatantly and easily noticed false information as answers.

It was clearly a simplified example, like I said endless bikeshed.

Here is a real one. I was using the much lauded new Gemini 3? last week and wanted it to do something a slightly specific way for reasons. I told it specifically and added it to the instructions. DO NOT USE FUNCTION ABC.

It immediately used FUNCTION ABC. I asked it to read back its instructions to me. It confirmed what I put there. So I asked it again to change it to another function. It told me that FUNCTION ABC was not in the code, even though it was clearly right there in the code.

I did a bit more prodding and it adamantly insisted that the code it generated did not exist, again and again and again. Yes I tried reversing to USE FUNCTION XYZ. Still wanted to use ABC

what i enjoy the most is every "AI will replace engineers" article is written by an employee working at an AI company with testimonials from other people also working at AI companies

TBH a lot of this is subjective. Including productivity.

My other gripe too is productivity is only one aspect of software engineering. You also need to look at tech debt introduced and other aspects of quality.

Productivity also takes many forms so it's not super easy to quantify.

Finally... software engineers are far from being created equal. VERY big difference in what someone doing CRUD apps for a small web dev shop does vs. eg; an infra engineer in big tech.

This is not always the case, but I get the impression that many of them are paid shills, astroturf accounts, bots, etc. Including on HN. Big AI is running on an absurd amount of capital and they're definitely using that capital to keep the hype cycle going as long as possible while they figure out how to turn a profit (or find an exit, if you're cynical - which I am).

That’s a bit of a reductive view.

For example, even the people with the most negative view on AI don’t let candidates use AI during interviews.

You can disagree on the effectiveness of the tools but this fact alone suggests that they are quite useful, no?

There is a difference between being useful for sandboxed toy problems and being useful in production.

Not really. I'd rather find out very quickly that someone doesn't know a domain space rather than having to wade through plausible looking but bad answers to figure out the exact same thing.

At this point it's foolish to assume otherwise. Applies to also places like reddit and X, there are intelligence services and companies with armies of bot accounts. Modern LLM makes it so easy to create content that looks real enough. Manufacturing consent is very easy now.

It's because the thing is overhyped and too many people are vested in keeping the hype going. Facing reality at this point, while necessary, is tough. The amount of ads for scam degrees from reputable unis about 'Chief AI Officer' bullshit positions is staggering. There's just tooo much AI bubbling

If someone seems to have productivity gains when using an AI, it is hard to come up with an alternate explanation for why they did.

If someone sees no productivity gains when using an AI (or a productivity decrease), it is easy to come up with ways it might have happened that weren't related to the AI.

This is an inherent imbalance in the claims, even if we both people have brought 100% proof of there specific claims.

A single instance of something doing X is proof of the claim that something can do X, but no amount of instances of something not doing X is proof of the claim that something cannot do X. (Note, this is different from people claiming that something always does X, as one counter example is enough to disprove that.)

Same issue in math with the difference between proving a conjecture is sometimes true and proving it is never true. Only one of these can be proven by examples (and only a single example is needed). The other can't be proven even by millions of examples.

I think it's a complex discussion because there's a whole bundle of new capabilities, the largest one arguably being that you can build a conversational interface to any piece of software. There's tons of pressure to express this in terms of productivity, financial and business benefits, but like with a coding agent, the main win for me is reduction of cognitive load, not an obvious "now the work gets done 50% faster so corporate can cut half the dev team."

I can talk through a possible code change with it which is just a natural, easy and human way to work, our brains evolved to talk and figure things out in a conversation. The jury is out on how much this actually speeds things up or translates into a cost savings. But it reduces cognitive load.

We're still stuck in a mindset where we pretend knowledge workers are factory workers and they can sit there for 8 hours producing consistently with their brain turned off. "A couple hours a day of serious focus at best" is closer to the reality, so a LLM can turn the other half of the day into something more useful maybe?

There is also the problem that any LLM provider can and absolutely will enshittify the LLM overnight if they think it's in their best interest (feels like OpenAI has already done this).

My extremely casual observations on whatever research I've seen talked about has suggested that maybe with high quality AI tools you can get work done 10-20% faster? But you don't have to think quite as hard, which is where I feel the real benefit is.

There are different types of contrary claims though, which may be an issue here.

One example: "agents are not doing well with code in languages/frameworks which have many recent large and incompatible changes like SwiftUI" - me: that's a valid issue that can be slightly controlled for with project setup, but still largely unsolved, we could discuss the details.

Another example: "coding agents can't think and just hallucinate code" - me: lol, my shipped production code doesn't care, bring some real examples of how you use agents if they don't work for you.

There's a lot of the second type on HN.

Yeah but there's also a lot of "lol, my shipped production code doesn't care" type comments with zero info about the type of code you're talking about, the scale, and longer term effects on quality, maintainability, and lack of expertise that using agentic tools can have.

That's also far from helpful or particularly meaningful.

There's a lot of "here's how agents work for me" content out there already. From popular examples from simonw and longer videos from Theo, to thousands of posts and comments from random engineers. There's really not much that's worth adding anymore. (Unless you discover something actually new) It works for use cases which many have already described.

Using two randos that are basically social media personalities as SMEs is just a damning statement about the current trends of programming.

The area is still relatively fresh. Those two media personalities do actual work though and provide a summary for today's state. You can wait for an academic research on what happened 6 months ago or a consulting industry keynote/advertisement about what they implemented a year ago... but I'm not sure you'll be better informed.

… and trollish to boot. Y U gotta “lol”?

But since there’s grey in my beard, I’ve seen it several times: in every technological move forward there are obnoxious hype merchants, reactionary status quo defenders, and then the rest of us doing our best to muddle through,

> Y U gotta “lol”?

Because some opinions are lazy. You can get all the summaries you want by searching "how I use agentic coding / Claude code" on the web or similar queries on YouTube, explaining in lots of details what's good and bad. If someone says "it's just hallucinations", it means they aren't actually interested and just want to complain.

Last time I ran into this it was a difference of how the person used the AI, they weren't even using the agents, they were complaining that the AI didn't do everything in one shot in the browser. You have to figure out how people are using the models, because everyone was using AI in browser in the beginning, and a lot of people are still using it that way. Those of us praising the agents are using things like Claude Code. There is a night and day difference in how you use it.

As a CS student who kinda knows how to build things. I do in fact get a speedup when querying AI or letting AI do some coding for me. However, I have a poor understanding of the system it builds, and it does a quite frankly terrible job with project architecture. I use Claude sonnet 4.5 with Claude code, and I can get things implemented rather quickly while using it, but if anything goes wrong I just don’t have that great of an idea where anything is, what code is in charge of what, etc. I can also deeply feel the brainrot of using AI. I get lazy and I can feel myself getting worse at solving what should be easy problems. My mental image of the problem to solve gets fuzzy and I don’t train that muscle like I would if I didn’t use AI to help me solve it.

>> when others make claims to the contrary suddenly there is some overwhelming burden of proof that has to be reached

That is just plain narcissism. People seeking attention in the slipstream of megatrends, make claims that have very little substance. When they are confronted with rational argument, they can’t respond intellectually, they try to dominate the discussion by asking for overwhelming burden of proof, while their position remains underwhelming.

LinkedIn and Medium are densely concentrated with this sort of content. It’s all for the likes.

> One thing I find really funny is when AI enthusiasts make claims about agents and their own productivity its always entirely anecdotally based on their own subjective experience, but when others make claims to the contrary suddenly there is some overwhelming burden of proof that has to be reached in order to make any sort of claims regarding the capabilities of AI workflows. So which is it?

Really? It's little more than "I am right and you are wrong."

Public discourse on this is a dumpster fire. But you're not making a meaningful contribution.

It is the equivalence of saying: Stenotype enthusiasts claim they're productive, but when we give them to a large group of typers we get data disproving that.

Which should immediately highlights the issue.

As long as these discussions aren't prefaced with the metric and methodology, any discussion on this is just meaningless online flame wars / vibe checks.

subjective experience is heavily influenced by expectations and desires, so they should try to verify.

Everything you need to know about AI productivity is shown in this first chart here:

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

Not confident it's quite that straightforward. Here's a presentation from Meta showing a 6-12% increase in diff throughput for above-median users of agentic coding: https://www.youtube.com/watch?v=1OzxYK2-qsI

Which is it is clear - the enthusiast have spent countless hours learning/configuring/adjusting, figuring out limitations, guarding against issue etc etc etc and now do 50 to 100 PRs per week like Boris

Others … need to roll up the sleeves and catch up

There isn't anything clear until someone manages to publish measurable and reproducible results for these tools while working on real world use cases.

Until then it's just people pulling the lever on a black box.

Hundreds of millions of people use these every day on real world use cases. If they didn’t work, people wouldn’t use them.

This is the measurable evidence you are talking about: https://a16z.com/revenue-benchmarks-ai-apps/

Here's even earlier evidence: https://en.wikipedia.org/wiki/Tulip_mania

this point is about revenue not valuations

All three companies profiled are selling products TO vibe-coders, not apps created BY AI-utilizers.

The shovel seller in the gold rush analogy.

Merely counting PRs is not very impressive to me. My pre LLM average is around 50/week anyway. But I’m not going to claim that somehow makes me the best programmer ever. I’m sure someone with 1 super valuable PR can easily create more value than I do.

Maybe I'm just in a weird place, but I can't imagine 50 PRs a week.

Maybe it's because I spend a lot of my time just turning problem reports reports on slack into tickets with tables of results and stack traces.

automate that shit

Unfortunately its mostly B2B integration stuff, where the other end is another company which can sometimes be just as a quirky as a user, except at scale.

"I received your spreadsheet detailing 821 records that are in State A but still haven't been moved to State B by our system as it adds Datapoint X on a regular basis. From what I can tell, it seems your data is missing crucial pieces you assured us would always be there. What's that? You want us to somehow fix whatever is somehow making those records in your AcmeERP system? Don't you have a support contract with that giant vendor? We seem like an easier target to hit up for impromptu tech-support consulting work? Well, I'll escalate that to the product manager..."

A bunch of tiny PRs is not hard to do manually. But LLMs can write boatloads of code to do kind of sophisticated things. You do have to figure out how to get to a point where you can trust the code. But the LLMs can help you write boatloads of tests too based on plain English descriptions.

llms remove a lot of the difficulty of writing a ton of reasonable code, but is that really the bottleneck to producing a bunch of PRs?

isn't it the reviewing time? reviewing code is hard work

Reviewing code can be hard but it's not as hard as writing the code. Even with the best autocomplete, and ergonomic editors like vim, it still takes quite a bit of time to write code for some features compared to the actual concepts being implemented. There are also lots of decisions like variable names that can be automated with a LLM. If you don't like what it came up with, you can tell it to change them. I recommend that you keep them fairly unique like you would for your own handwritten code, because ambiguity creates problems for people and machines alike.

For me, review is the hard part.

Or the tool makers could just make better tools. I'm in that camp, I say make the tool adapt to me. Computers are here to help humans, not the reverse.

so when you get a new computer you just use it, as-is, just like out of the box that’s your computer experience? you don’t install any programs, connect printer, nothing eh? too funny reading “tool should adapt to me” and there are roughly 8.3 billion “me” around - can’t even put together what that means honestly

People working in languages/libraries/codebases where LLMs aren't good is a thing. That doesn't mean they aren't good tools, or that those things won't be conquered by AI in short order.

I try to assume people who are trashing AI are just working in systems like that, rather than being bad at using AI, or worse, shit-talking the tech without really trying to get value out of it because they're ethically opposed to it.

A lot of strongly anti-AI people are really angry human beings (I suppose that holds for vehemently anti-<anything> people), which doesn't really help the case, it just comes off as old man shaking fist at clouds, except too young. The whole "microslop" thing came off as classless and bitter.

the microslop thing is largely just a backlash at ms jamming ai into every possible crevice of every program and service they offer with no real plan or goals other than "do more ai"

Reply on news.ycombinator.com