Comment by fullstackchris | Hacker News Reader

Comment by fullstackchris | Hacker News Reader

These are almost certainly hallucinated. Why would they use the prompt interface itself to try and 'protect' the software itself? How in the world do people think this is really "the golden guiding prompt, straight from the engineering team!". It _looks_ convincing because hey, _every_ 'answer' it spits out looks convincing, regardless what you ask it.

Imagine thinking that a list of 20 or so items is the 'software lock' on a billion parameter language model... literally any longer than one second of thinking can tell you what the real answer is.

There's no illumati stuff going on here, just people wanting it to be that way.

Keep in mind a lot of these "AI" Twitter guys (this guy included, definitely more of the management type, NOT software engineer. Yes I know he and his team won the Boring project blah blah blah, so let's see some code that you've written - oh, 5 whole repositories on GitHub, wow, none even remotely associated with machine learning) are just snake oil goons hopping (and profiting) from the current AI hype bubble who have never actually written code for, or trained a model, or even probably know the concepts behind training or building models anyway.

Getting really sick of clueless goons around this AI stuff. Educate yourself!

> These are almost certainly hallucinated.

You are almost certain of this based on the fact you can't think of a reason it would be done that way?

> There's no illumati stuff going on here, just people wanting it to be that way.

What are you talking about? Even if these rules were not hallucinated, nobody is suggesting "illumati[sic] stuff" going on. What are you actually thinking the implications would be if it were true?

The guy said he got the same doc word for word from several similar prompts, even down to the typo. That makes the hallucination hypothesis less likely.

There’s always the chance that he’s just lying though. I find that much more likely.

So what's the real answer buddy? Because there's no extra layers here. There's no magical brain making decisions.

Their choices are:

- Insert a prompt with rules so that the model is primed to produce an answer taking those into account (they do that, it's literally an option in the API to modify the rules prompt)

- Train the model so that when no-no words are said, it likes to produce a specific answer such as "I am a google product and I like google" instead of no-no thoughts

They use both of these techniques but neither are particularly effective.

Believe it or not, this is how most of these systems are built - when we talk about "prompt engineering" this is what we mean.

I wrote more about why I doubt this is a hallucination here: https://news.ycombinator.com/item?id=35925239

Reply on news.ycombinator.com