Comment by firasd | Hacker News Reader

Comment by firasd | Hacker News Reader

The problem is that when the LLM writes ad-hoc code how can you trust it?

For example in my Liveclip MCP server I'm working on (not released yet) I have table manipulation type tools

So for example a couple days ago Claude made a combined ranking score for my Youtube Shorts analytics -- it did all these tool calls in the same turn

First it normalized the % values into numbers

{ "destination_col": "G", "key": "cinemasleepstories_temp_2026071702", "pattern": "%", "replacement": "", "source": { "col": "E", "row_start": 1, "row_end": 6 } }

Then it made the composite score

{ "dest_col_start": "H", "dest_row_end": 6, "dest_row_start": 1, "expr": "round((likes+1)(followers+1)stayed, 1)", "source_key": "cinemasleepstories_temp_2026071702", "sources": [ { "col": "C", "name": "likes", "row_offset": 0 }, { "col": "D", "name": "followers", "row_offset": 0 }, { "col": "G", "name": "stayed", "row_offset": 0 } ] }

And then it set the new headers

{ "headers": { "G": "Stayed (numeric)", "H": "Composite Score" }, "key": "cinemasleepstories_temp_2026071702" }

If Claude just said "let me write some pandas against this CSV" the workflow would be a lot more iffy and generally uncomfortable/ephemeral

Mine is forced to always start with writing tests (unit, e2e, smoke), then the script (with eager/early fail modes). I have yet to see this approach to fail.

That's obviously a part of the broader picture Claude always runs inside bwrap sandbox, any work involving python/nodejs must happen in the dedicated devcontainer, standard operations are reused via Makefile, Ai can use dedicated account in the developed $thing for testing in live, hooks governing tool calling are brutal, but it works.

For me at least.

A python script is now more iffy than a LLM going through JSON?

Yeah, of course. The MCP server tools do use Python in the backend but just look at the affordances it provided in the args -- named columns, a fixed match/replace API, a way to specify the range of rows. The LLM is not gonna create a little library with these affordances in an ephemeral REPL it's just gonna YOLO some pandas code that you hope did what you wanted and addressed the right columns, had the right types, etc

I feel like I am having a fever dream. How is an LLM that is trained on endless examples of Python, a well defined language as well, going to generate worst code than some random JSON interface?

Have you ever tried analyzing a CSV in an LLM? Maybe then you'd get what I'm saying

Heck for a lot of questions the LLM won't even write Pandas and just try to eyeball the arithmetic and be like "Yeah March 3 to March 7 values add up to..."

And often -- without using code _openly_ or in thinking -- it is wrong because it is just a "guesstimate" at best due to the nature of how LLMs generate tokens.

The difference is a yolo’d script needs to be verified everytime it’s generated. So it makes sense to generate it and verify it once (a cli command or mcp call).

I think OP’s example is bad. As that (if I understand the example correctly as I don’t really know what liveclip is) sounds like something worth generating a script for once and reusing it. However, if you’re looking to book a flight, you would prefer an airline cli or mcp server or a nodejs script using puppeteer to automate the browser or even some airline sdk to buy a ticket?

What doesn’t make sense is generating a script to achieve a task without clear feedback loop for verifying the script correctness. In OPs example, there is no way for the LLM to very the correct final result since it’s unknown. It can “code review” the code looking for bugs, but bugs in scripts happen.

we have a way to see created scripts in a dashboard, and in fact they are not ad-hoc only, i.e. it can create "saved scripts" that it can re-use, iterate and so on.

Also not exec happens in a sandboxed env, which means it might not have access to the fs by default (it has to agent-fs, which is a layer we built on top of s3 for agents)

Yeah I guess the dichotomy between 'tools' and 'code mode' isn't really as clear especially if the tools themselves are intelligent and truncate long results and have start_range, end_range type arguments. So the gist is about 'what part of the output needs to reach the LLM' and a 'code mode' approach can help orchestrate that just like good tools can

exactly, I love how the cloudflare article puts it in fact

Reply on news.ycombinator.com