638 karma · joined April 7, 2020
We're modern day factory workers.
All code written by an LLM is reviewed by an additional LLM. Then I verify that review and get one of the agents to iterate on everything.
Just more FUD from devs that think they're artisans.
Because when I see people that are downplaying LLMs or the people describing their poor experiences it feels like they're trying to "vibe code" but they expect the LLM to automatically do EVERYTHING. They take it as a failure that they have to tell the LLM explicitly to do something a couple times. Or they take it as a problem that the LLM didn't "one shot" something.
Like people in here complaining about how poor the tests are... but did they start another agent to review the tests? Did they take that and iterate on the tests with multiple agents?
I can attest that the first pass of testing can often be shit. That's why you iterate.
"vibe coding" didn't really become real until 2025, so how were they vibe coding for 2 years? 2 years ago I couldn't count on an llm to output JSON consistently.
Overall the article/video are SUPER ambiguous and frankly worthless.
Another agent reviews the tests, finds duplicate code, finds poor testing patterns, looks for tests that are only following the "happy path", ensures logic is actually tested and that you're not wasting time testing things like getters and setters. That agent writes up a report.
Give that report back to the agent that wrote the test or spin up a new agent and feed the report to it.
Don't do all of this blindly, actually read the report to make sure the llm is on the right path. Repeat that one or two times.
Just like anything else in software, you have to iterate. The first pass is just to thread the needle.
In the US education has been bastardized into "job training"
Good workers don't really need to think in this paradigm.
There's been such a massive leap in capabilities since claude code came out, which was middle/end of 2025.
2 years ago I MAYBE used an LLM to take unstructured data and give me a json object of a specific structure. Only about 1 year ago did I start using llms for ANY type of coding and I would generally use snippets, not whole codebases. It wasn't until September when I started really leveraging the LLM for coding.
That plus the completion of high impact projects makes good strong engineers.
Those are the people I see using LLMs
What about git stats?
I can tell you the guys that are consistently pushing code AND having the biggest impact are using LLM tools.
Anyways, I replied because I had something else I wanted to say.
I was using Gemini in a google worksheet a while back. I had to cross reference a website and input stuff into a cell. I got Gemini to do it, had it do the first row, then the second, then I told it to do a batch of 10, then 20. It had a hiccup at 20, would take too long I guess. So I had it go back to 10. But then Gemini tells me it can't read my worksheet. I convince it that it can, but then it tells me it can't edit my worksheet. I argue with it, "you've been changing the worksheet wtf?" I convinced it that it could and it started again, but then after doing a couple it told me it couldn't again. We went back and forth a bit, I'd get it working, it would break, repeat. I think it was after the third time I just couldn't get it to do it again.
I looked up the docs, searched online, and I was concerned that I found Google didn't allow Gemini to do a lot of stuff to worksheets/docs/other google workspace stuff. They said they didn't allow it to do a ton of stuff that I definitely had Gemini doing.
Then a week or two went by and google announced they're allowing gemini to directly edit worksheets.
So wtf how did I get it to do it before it could do it???
Manage that yourself! If you have hooks throwing errors then feed the error back into the llm.
I'm personally not advocating for not having humans in the loop. I don't know of anybody using llm tools or advocating for them that are saying there shouldn't be humans in the loop. "vibe coding" seems to mean different things to different people.
I've had one llm one-shot a codebase. Then I use another one to review (with a pretty explicit prompt). I take that review and feed it to another agent to refactor. Repeat that a bunch of times.
I run multiple agents in separate sessions. It starts with one agent, building out features or working on a task/bug fix. Once it gets some progress, I spin up another session and have it just review the code. I explicitly tell it things to look out for. I tell it to let me know about things I'm not thinking of and to make me aware of any blind spots. Whatever it reviews I send back to the agent building out features (I used to also review what the review agent told me about, but now I probably only review it like 20% of the time). I'll also have an agent session started just for writing tests, I tell it to look at the code and see if it's testable, find duplicate code, stale/dead code. And so on and so forth.
Between all of that + deterministic testing it's hard for shit to end up in the code base.
I'm definitely faster, but there's a lot of LLM overhead to get things done right. I think if you're just using a single agent/session you're missing out on some of the speed gains.
I think a lot of the gains I get using an LLM is because I can have the multiple different agent sessions work on different projects at the same time.
The assumption is that your test is right. That's TDD. Then you write your code to conform to the tests. Otherwise what's the point of the tests if you're just trying to rewrite them until they pass?
I literally just posted in another comment that people shouldn't be worried about killing their current session/context window. I used to get worried about compaction and losing context, but now when I feel like things are slipping I kill it quick and start a new session.
If anything the silver bullet people are mostly managers and C levels... some of which don't even use the tools themselves.
Of the devs that rejected it at first, the ones with the same sentiment I'm seeing online in threads like these, we forced one to give it a try. He now treats totters between using it well and treating it as a silver bullet. I still hear him incredulous about the things claude does at meetings, "I had to do <thing> and I thought I'd let claude get a crack at it... did it in one shot"
I wonder if some people are putting in too much into their markdown files of what NOT to do.
I hate people saying the llms are just better auto-correct, but in some ways they're right. I think putting in too much "don't do this" is leading the llm down the path to do "this" because you mentioned it at all. The LLM is probabilistically generating it's response based on what you've said and what's in the markdown files, the fact you put some of that stuff in there at all probably increases the probability those things will show up.
For the llm a lot of linting and build/test tools go into simple scripts that the llm can run and get shorthand info out of. Some tools, if you have the llm run them, they're going to ingest a lot from the output (like a big stacktrace or something). I want to keep context clean so I have the llm create the tool to use for build/test/linting and I tell it to create it so the outputs will keep its context clean, then I have it document it in the .md file.
When working with the LLM I have to start out pretty explicit about using the tooling. As we work through things it will start to automatically run the tooling. Sometimes it will want to do something else, I just nudge it back to use the tooling (or I'll ask it why or if there are benefits to the other way and if there are we'll rebuild the tooling to use the other way).
Finally, if the LLM is really having trouble, I kill the session and start a new one. It used to feel bad to do that. I'd feel like I'm losing a lot of info that's in context. But now, I feel like it's not so bad... but I'm not sure if that's because the llms are better or if my workflow has adapted.
Now, let me backup a little bit. I mentioned that I don't have the llm use git. That's the control I maintain. And with that my workflow is: llm builds feature->llm runs linters/tests->I e2e test whatever I'm building by deploying to a dev/staging/local env->once verified I commit. Now I will continue that context window/session until I feel like the llm starts fucking up. Then I kill the session and start a new one. I rarely compact, but it does happen and I generally don't fret about it too much.
I try to keep my units of work small and I feel like it does the best when I do. But then I often find myself surprised at how much it can do from a single prompt, so idk. I do understand some of the skepticism because a lot of this stuff sounds "hand-wavy". I'm hoping we all start to hone in on some general more concrete patterns but with it being so non-deterministic I'm not sure if we will. It feels like everyone is using it differently and people are having successes and failures across different things. People where I work LOVE MCPs but I can't stand them. When I use them it always feels like I have to remind the llm that it has an MCP, then it feels like the MCP takes too much context window and sometimes the llm still trips over how to use it.
I messed around with separate "agents" in the same context window for a while. I even went as far as playing with strands agents. Having multiple agents was a crapshoot.
Sometimes they'd work great, but sometimes they start working on the same files at the same time, argue with each other, etc. I'd always get multiple agents working, at least how I assumed they should work, by telling the llm explicitly what agents to create and what work to pass off to what agents. And it did a pretty poor job of that. I tried having orchestration agents, but at a certain point the orchestration agent would just takeover and do everything. So I'm not big on having multiple agents (in theory it sounds great, especially since they are supposed to each have their own context window). When I attempted doing this kind of stuff with strands agents it honestly felt like I was trying to recreate claude, so I just stick with plain cli llm tools for now.
When I work on issues I create a new branch off of master, let the llm go to town on it, then I manually commit and push to remote for an MR/PR. If there are any errors on the commit hooks I just feed the errors back into the agent.