Hacker News Reader

You should be careful. It’s not just about being faster. LLMs can write more tests, more reliably, and find things that must be changed following a change that you might overlook, and find contradicting requirements so much better than humans. You can probably keep up in terms of speed , though I doubt it for nearly every programmer I’ve ever met including myself… but almost certainly not with the same level of quality and thoroughness, despite what people on HN seem to think.

> LLMs can write more tests

In fact, sometimes LLMs write too many tests to the point it significantly slows down your test runs and CI!

Too many tests and hallucinates edge cases and “security” flaws.

Good point. Might be good on a weekly basis to purge / look for stale and unnecessary tests.

Shouldn’t that be caught at review time? Or by the sad little ai driver that submits the pr? ;)

I had that issue so I told it to refactor the tests to a smaller more concentrated significant set.

I’ll be ok, I produce higher quality work than any LLM ever will. And if that time ever comes, which I seriously doubt, it’s not like it’s that hard to prompt.

LLMs will hallucinate edge cases or worry about things that just aren’t possible within context. It results in more tests than are needed. I personally dont think LLMs are any good at all.

I can’t believe I’m writing this comment.

A month ago I was still talking like you. Then I said to an insisting colleague, “watch I’ll try to vibe code a game engine and show you the crap it produces.”

Granted, this wasn’t my first game engine so I knew exactly what to say, but I was completely floored. GPT Sol & Astra at max effort was flawless at nearly everything I threw at it. I’m talking fully ground up, zero dependencies, just the primitives and SIMD. Fully featured engine done in two weeks with deferred 2-stage render pipeline, shadow maps, mesh shaders, MSAA, post processing, collision, physics, the works.

I even intentionally skipped a few important optimizations and went back to refactor them in, thinking there’s no way it can do a wholesale rewrite of major systems, but it did it. I got dry mouth from all the jaw dropping. My prompts got shorter and more ambiguous, so it would ask me clarify. I audited every line of code, and it was good (after adding two skills.md). I gave up. I’m a reluctant believer.

It sucks, but sadly these things are really good. Don’t be last contrarian, there’s nothing to gain. You’re just lying to yourself

Have you tried creating something that doesnt have HUNDREDS of existing implementations on the Web already?

Yes of course. It’s trivially easy to say “procedurally generate a mesh of description of unique shape”. If you can put it into words, the LLM can do it. Do you have any examples of something it can’t do?

I think this is the part that stumps me. I've done similar things, I'm not math heavy but "vibecoded" a full physics simulator. It wasn't 100%, but it was 90% or 95%. Wild!

But I dropped it and will probably never touch it again.

Are you going to do anything with that game engine? I'm sure there are thousands of others who've written a game engine with the tools. What are they being used for? Is it just to "try it out"? I see people producing a LOT of software. Sure, the code is good, but was it needed? Are people using it?

And maybe that's OK - we're just writing software for ourselves now, because it's so cheap + fast to produce.

> Are you going to do anything with that game engine? I'm sure there are thousands of others who've written a game engine with the tools. What are they being used for? Is it just to "try it out"? I see people producing a LOT of software. Sure, the code is good, but was it needed? Are people using it?

You can ask the same about hand written code too.

Maybe I’m just better than you? It’s something to consider. Your whole point here is that astra could pump out something faster than you ever could at a quality level _you_ think is good. It might be worth some reflection that _your quality level_ is not _my quality level_.

I have no pressure to use this stuff. I’m still getting paid. I’m still shipping higher quality software at the same speed as the rest of my peers. Why should I sacrifice quality for speed?

I'm quite curious to know what kind of systems / software do you work on?

You just described the exact same scenario that I’ve heard over and over for the last 2 years. “I didn’t think AI would take over software engineering. But recently, it blew me away with a greenfield implementation of something that already exists and that I know how to make. It has few users and has spent little time in the wild. Now I’m a believer.”

It’s definitely getting better, but this isn’t the kind of thing that is going to change the world. It needs to be good at things that are much more entangled with the world for that.

Not the argument I was making and I’m not disagreeing with you.

And yet LLMs still seem to write the same kind of crap brittle mock-the-world tests that average developers do, drives me nuts.

Of course. LLMs only produce the average. Most of the time if you think you need to mock you should probably just do the real thing. It’s mostly an antipattern.

We improved that by creating a skill that is precise about what we want our tests to look like, and telling the AI to always use the skill when writing tests. It still writes a bit too many and uses a bit too much mocking, but the alternative of humans missing obvious use cases and just being lazy to test important cases is much, much worse.

People say this, and it sounds reasonable. But, as a user of software, I certainly am not feeling a lot of software getting more reliable and higher quality. Mostly I feel the opposite of that.

This is exactly the approach I’m limiting myself to: LLMs as a very smart rubber duck to assist in research and understanding complex systems. I have no interest in it writing the code for me, just like I have no interest in another person doing the same.

I am the engineer, I am the one that has the vision, and I am the one that needs to understand the thing down to the minute details.

That said, most systems I still write are not so complex I need a very smart assistant, so I rarely use LLMs at all; it is still a valuable skill being able to research and think hard for yourself. An LLM cannot think out of the box (of its training dataset), and there lie the discoveries and paradigm shifts that move tech forward.

It’s nice to see some sanity in an insane world.

I really and unironically applaud you, when the code you're writing has higher quality than the average AI produces AND it means something.

In the domains I get paid to work in, it simply doesn't matter.

The code was shit to begin with, because of hundreds of hacks due to underspecified or simply wrong requirements, bad code practices, architecture that couldn't keep up but was never fixed due to stubbornness...

Not saying we humans would do better or worse in general, just sharing my experiences.

I have customers where AI usage is absolutely forbidden and others where it's totally fine and colleagues vibecoded mess will come to bite them/us all.

I enjoy programming and producing high quality work. Why would I want to give up the thing I find fun and produce something subpar?

> The code was shit to begin with, because of hundreds of hacks due to underspecified or simply wrong requirements, bad code practices, architecture that couldn't keep up but was never fixed due to stubbornness...

My coworkers have always produced slop, even before LLMs. Unfortunately, as the lead on the project it’s still my responsibility to get that up to a certain quality level or at least make it isolated and malleable enough that it can be changed and we all won’t have a bad time doing it.

I guess what I’m saying is.. I get it. It’s hard to work with other people who are just pushing whatever is given by the LLM. But I still have fun doing it myself.

I also still enjoy it. In my spare time.

I've always enjoyed it. In my spare time.

Also I wouldn't go as far to say that I myself produce higher quality code than my peers.

I am between a 5/10 and a 7/10 programmer at best.

I bring other valuable skills though.

> I enjoy programming and producing high quality work. Why would I want to give up the thing I find fun and produce something subpar?

No one who cares wants to. That's not the question.

The question is whether you can continue to get paid doing so.

I seem to be continuing to get paid. What’s your next excuse about why you’re sacrificing quality?

My question wasn't one for right now, it's an ongoing question as more of the field starts adopting AI.

I could keep pace until about half a year ago. But it meant hands-on typing code for the full work day. Now with claude I can produce similar output in 1-2 hours, most of the time is spent on iterating and having the LLM review the code and fix it by itself and then me testing everything. There is more risk of letting the LLM make judgements and building the wrong thing and me having a smaller chance of discovering holes in the requirements because claude is happy to confidently make the wrong decisions.