> This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code.
I agree, yes. But it's a bit like saying the only way to not die in a car crash is to not drive. If we're in a situation where using LLMs is unavoidable, I would rather make them safer.
> Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust the model - and you cannot - there is always the issue of prompt injection. A proper safeguard means actual sandboxing.
You're right. I did not completely sandbox it and air gapped it. But I also wented it to do some actual work.
If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.
The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.
But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.