eithed

Don't forget that LLMs are trained on human code. If they cannot understand what your code does then they cannot make changes to it, or at least - having them understand your codebase becomes expensive (more trips to Anthropic servers)

I disagree. What I should have added is that with agents (as well as humans) you do need to have tests that verify what was done.

I mirror your thoughts. I think we'll end up with "perfect map" paradox = you cannot be vague or indecisive on what you want (and if you are then these decisions don't matter) and you're creating a 1:1 representation of what the code needs to be.

I'd substitute "owner" for the team and in that sense the owner will not need to be human.

We're at this state where Claude is great at doing the "middle" part of work, but it's crap at gathering requirements and verification of what it has done. I also don't see people caring about these aspects of software development as shown in the article

Exactly - the normalization of craft (?) is interesting

Don't forget that you can adjust your requirements (either via plan or skill) to ensure the mistakes do not happen. The problem is that neither LLMs, nor humans (that don't work with the domain) will know they made these mistakes. Even coders don't think about everything all the time

The road will be built to some specs, including features nobody asked for. If the corpus was trained for roads built in Arctic, you will get penguin crossings.

I agree. But if I'm paying for the road (even as a taxpayer) I get angry that after a year it's full of potholes and that there are unnecessary signs warning about penguin crossing, making it cost 2 times more than it should have (and dont get me started why this road is really a highway leading to my house). I'd want certain qualities. And this article is basically = you will get a road, built quickly

But yes, you are right - I don't build roads and don't know what is a price to build a road and how to determine the quality of correctly built one, nor I will ever care or learn.

True, but you should say that about every thing. Does it matter to you how the car drives, as long as it takes you to your destination? Well, yes, it matters: how will it deal with a crash, and if it's possible to replace a part and if anybody can just open it if you leave it outside. I will be amazed if somebody shows me their home-printed car, but if they'll try to sell it to me like a new one...

Fair enough, but enterpreunership should, I guess, ask questions if given Next Big Thing has substance behind it or is it just snake oil.

What I find fascinating that there is so little substance in this article about the quality of produced code and the medium. Is the code documented and tested? Is it understandable and extendable? Is it secure? What language, framework, database was used? Author mentions judgement and taste - well, is the code tasteful? Will the model rearchitecture the entire thing if I ask it to add new functionality, spending another 9.5h in tokens? I assume that the research part is domain knowledge = how different types of travel translate to time making it presentable; how did the author verify this?

These questions are even not about AI: if I were to give money to a human agency and were given something they tell me works, I would ask the same questions. If I did not know how to evaluate, I would hire people that do. With LLMs the verification part is what bothers me the most.

I assumed that as it was hitting too close home (both PMs and EMs were expected to use AI, with PM trying to code in solutions that they didn't have expertise nor domain knowledge to deal with; EM was prototyping solutions that had access to prod DB that were shut down as soon as we found out). I can only symphatize nonetheless and thank you for giving your PoV.

Even if it's a mandate top-down, you can:

- show increase in errors and outages caused by this approach

- integrate manager changes into your CI pipeline (coding / reviewing / testing / documentation)

- discuss how your manager can do the changes they need to do without sidetracking all other work

Make it indeed about the money: coding by PM + fixing what was coded + dealing with fallout is greater expense than coding by PM + automated guidelines + reviewing what was coded.

That is - if the environment you're working in is reasonable and it's not a power play by your PM

> Please write a manual on how to cleanup after AI rockstar managers who think they can code.

Why are you allowing AI rockstar managers to (I assume) push without code review? Why are you cleaning up the fallout? It's not AI issue, it's people issue

Sure, producing code has become cheap. Yet again the taste matters and LLMs do not have taste - they will apply patterns that are unnecessary or not extendible, producing unmaintainable systems that nobody understands. Capturing domain knowledge was the crux of development process, but so was verifying, documenting, ensuring that multiple systems work together, maintaining uniformity. I don't know where the assumptions, done by developers, that they only need to produce code that just works or goes brrr fast comes from.

Domain expert can develop working code, but they will not be able to ensure above.

Depends - using Sonnet here and generally it should be as you say: plan would produce the result.

Still Claude will sneak things in - in my recent plan, for example I had defined, per acceptance criteria what colours the statuses should be: green for live, blue for sold, grey for anything else; it changed this to: green for live, orange for in progress, blue for sold, red in demolition, etc. When pressed why did it to this, it was unable to explain why. This is with a plan where AC were explicitly provided from the task in Given/When/Then format and were to be adhered to strictly. I've caught this within planning, but I shouldn't need to be doing this.

Even in standard prompts where I tell it "Change this label from X to Y", it ended reordering the tabs unrelated to ask. Again I was not able for it to explain why - it was so abrupt. And it was in fresh context, without any pollution on what I expect it to do.

I also noticed a different behaviour regarding skill; today and yesterday it would not be following skill guidance at all ie: skill writing skill - I'd have to explicitly tell it to test skills after writing them, when this is a behaviour expected by default. Similarly with other skills - knowing that it should have done something per skill guidelines and it not doing it at all. This is new behaviour that I've not seen a week ago.

Can you explain the benefits of running this over rector / eslint? (and to certain degree phpstan / deptrac)

Write a skill outlining your expectations of the code, put that skill into the pipeline, so that it can be included within your workflow.

Webdev here, but currently I have: - a skill where I outlined how the architecture of the system should look like, with guards (static analysis, architecture tests, linting) confirming that the code it generates adheres to standards

- a skill that tells it how tests should look like (use generators, write both feature / unit tests)

- a skill that tells it to generate docs from the code in a form of acceptance criteria (Given / When / Then)

- a skill that tells it to generate frontend uat tests + accompanying backend seeders given the AC

- a skill that tells it to verify that ticket objectives match what was delivered

At this point I still need to guide it to move task from one stage to the other (coding, testing, verification that indeed what was coded adheres to what was required), but I believe that these dynamic workflows can automate this work as well.

Fair enough, I was looking at budget fields on wiki rather than reading the tidbits - thanks for the correction still!

True - your comment reminded me of Cube; that was done in 3 weeks, with budget of $350,000 CAD (according to wikipedia). Another favorite of mine - Primer = 5 weeks with budget of $7k.

edit: looking at others, Pi - 4w and $130k

While $500k 90m movie done in two weeks is an accomplishment, looking at the trailer it's very dubious to me on the quality of it. Plot, characters, audio - everything screams "I've already seen this somewhere", there's no substance here, at least for me. And while computer visuals are nice, it's nowhere "Love Sex Robots" quality where they're driven by computer graphics as well.

I agree and this response was following OPs example. But the point still stands - the goal is to outsource, in a weird way, the results being served = Google as such wouldn't need to pay for content. Now, if accuracy of such sources doesn't matter (or is good enough) for casual user...

I'd say it depends - evaluate the vibes. I spent 8y and recently 7y at a company where I genuinely responded with what I thought. But I'd say it's a matter of the audience - some people want to hear certain things and deciding if you can share these thoughts is up to you. It also allowed me to make decisions - if people don't care what I think and want sycophancy is this the company I want to be working at? I understand though it depends on one circumstances = you have to grin and bear it

Why can't people own to their mistakes and reschedule

This video tells me otherwise: https://www.youtube.com/watch?v=UDQds7VZkfg ( Cold Ones - We Drank AI's Horrible Cocktail Ideas). This is a tongue in cheek response though, as LLMs improved significantly since then.

Yeah, this - interview is a performance after all

> Why let a cooking website get visitors and ad revenue when they are free to take the content and show it as their own?

I think this is a step beyond that - why should people be creating cooking websites when you can ask LLM how to cook given thing, while indeed, serving their own ads. It's the continuation of "we own content other people produce" policy

Exactly! If you're an owner (ie: expert, you teach other people how to do your stuff) you should be making decisions and taking responsibility, given existing context. I'm happy using LLM to confirm my reasoning or research, but it's still me doing the coding, or architecting or anything, not LLM, and if something goes bad I cannot say "LLM told me to do it". If people are blindly doing what their tools tell them to do, that's the problem there.

Edit: in this instance if I were the expert I'd respond from my expertise. Using LLM is fine to explain whys/research per what you say, but ultimately I'm the educator here

I was working in an env where I started to suspect that people I was chatting with were using LLMs. These were people that didn't want to talk to me either way, so there was not much lost here. I suspected that, because the technical expertise they were showcasing when responding to messages would evaporate when talkin f2f

Yes :D

In Poland you would just say Hej! = I acknowledge you being there but I don't want to chat. But again, back to the point I was trying to make - the intent behind these sort of questions (and answers) can be cultural

Brits will either say "Fine, thanks" or not respond at all. Poles will tell you that yesterday they went to visit their relatives, and now they're tired, or that their cat is sick, or that they having problems with something.

It's a difference between "asking when you don't care about response" vs "asking because you genuinely want to know how the other person is"