Hacker News Reader

Comment by juliansark | Hacker News Reader

Guy gives non-deterministic software root access, desaster happens. Movie at eleven.

Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to know: total and absolute failure of guard rails, but these guard rails can not be expected to be in an LLM.

View on HN

Exactly.

Prompts are just weights on a graph traversal. They don't guarantee anything. The LLM does not "understand" the prompts and so it cannot fully adhere to them. They only improve the liklihood it will output what you want.

Never ever ever give an LLM access to something you can't afford to break. And stop thinking of them like people.

This feels like what a dog does. It's incredibly hard to train dogs by punishment, because it's very hard to tell if the dog understands what he did wrong and feels genuine remorse, or is just showing submissive signs at your display of dominance.

>total and absolute failure of guard rails

It seems here the guard rails at failure were the llm users right? Whatever guard rails you can think may be useless against the superior human stupidity.

Also, what's the LLM use policy at the SD-6?

> Guy gives non-deterministic software root access, desaster happens.

I agree the guy is an idiot for trusting these AI models.

OTOH AI companies keep running and marketing their services with zero accountability for mistakes.

I guess people are finding out the hard way you do sorta need technical people to say, "hey, maybe this isn't a great idea" rather than trusting marketing hype that says technical skills are dead.

I wonder, how should an AI company be accountable for non-deterministic nature of AI, which is a fundamental property of the said AI?

People have been drinking too much hopium they have lost touch with reality.

Everyone needs to properly understand these tools before they use them for anything serious.

At the very least, when an agent can delete a production database you should get an obvious warning whenever you enable it. Marketing wouldn't like it though.

He didn't give it root access, it found root access.

Reply on news.ycombinator.com