This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
I had a team proudly show me how they were feeding their dashboard data into an LLM to predict growth needs, instead of doing basic linear regression or any number of pre-existing basic statistical analysis. The part that depressed me the most on that was it was a bunch of engineers that did it, the people you'd most hope would understand what an LLM is/does.
You can go “here’s a spreadsheet of some data, do a thing to it” and it’ll go through, have it plan out the type of calculations you want to do, and it’ll write you a program to parse your sheet, calculate what you want, then output an result sheet.
The key to good results with minimal effort seems to be the planning modes to me, combined with the /goal command. A little bit of validation before implementation goes a long way.
I'm mostly starting to have some success telling TPMs and engineers to ask chatgpt to give them appropriate code that generates more accurate answers.
However sometimes I think that AI use shows a revealed preference around things. In this case, the actual data just ... maybe never mattered to them at all? People wanted a shiny chart and they got a shiny chart.
Today the problem you're mentioning is easily avoided if you are aware by ensuring you direct the agents to build tools for anything deterministic.
Every week "we" (myself and Claude) start by analysing revenue from the previous week. Claude will do some manual proposals but they will be based on analysis done by a deterministic script. It will then run workflows that are increasingly calling deterministic scripts - every week another few decision points the agent used to do (and that I had to then manually vet) gets replaced by deterministic scripts.
It is what allows what is a very low margin opportunity to work - it is scaling with the total amount of manual labour and total token use remaining near constant. It's an opportunity that I won't go into much detail about because the margins are thin and the moat narrow and it's basically only viable because of this approach of setting a tiny budget where ~1/3 of it or so is dedicated to systematically eliminating manual and LLM work from the process.
Even so, I expect the opportunity to actually extract profit from it is going to be short-lived - the moat will eventually evaporate if LLMs keep getting cheaper and better in favour of people "just" asking an LLM to do the whole thing end to end.
So on one hand I agree with you - there are lots of things that'd be better automated with scripts, and where there might well always continue to be efficiencies you can extract by turning things into deterministic scripts instead. On the other hand, LLMs are already so cheap for a lot of things that it increasingly doesn't matter, and good enough that where it does matter they will increasingly also just write tools themselves if you ask them and sometimes - and likely increasingly so over time - without even being asked.
That being said, the correct way would be to just use the LLM to produce the script, validate it themselves (often times there's something very minor wrong), and then use that script next time instead of an LLM to rewrite the same script over and over every time you need to produce a report.
For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.
(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)