LLMs will hallucinate edge cases or worry about things that just aren’t possible within context. It results in more tests than are needed. I personally dont think LLMs are any good at all.
A month ago I was still talking like you. Then I said to an insisting colleague, “watch I’ll try to vibe code a game engine and show you the crap it produces.”
Granted, this wasn’t my first game engine so I knew exactly what to say, but I was completely floored. GPT Sol & Astra at max effort was flawless at nearly everything I threw at it. I’m talking fully ground up, zero dependencies, just the primitives and SIMD. Fully featured engine done in two weeks with deferred 2-stage render pipeline, shadow maps, mesh shaders, MSAA, post processing, collision, physics, the works.
I even intentionally skipped a few important optimizations and went back to refactor them in, thinking there’s no way it can do a wholesale rewrite of major systems, but it did it. I got dry mouth from all the jaw dropping. My prompts got shorter and more ambiguous, so it would ask me clarify. I audited every line of code, and it was good (after adding two skills.md). I gave up. I’m a reluctant believer.
It sucks, but sadly these things are really good. Don’t be last contrarian, there’s nothing to gain. You’re just lying to yourself
But I dropped it and will probably never touch it again.
Are you going to do anything with that game engine? I'm sure there are thousands of others who've written a game engine with the tools. What are they being used for? Is it just to "try it out"? I see people producing a LOT of software. Sure, the code is good, but was it needed? Are people using it?
And maybe that's OK - we're just writing software for ourselves now, because it's so cheap + fast to produce.
You can ask the same about hand written code too.
I have no pressure to use this stuff. I’m still getting paid. I’m still shipping higher quality software at the same speed as the rest of my peers. Why should I sacrifice quality for speed?
It’s definitely getting better, but this isn’t the kind of thing that is going to change the world. It needs to be good at things that are much more entangled with the world for that.