Hacker News Reader

Comment by Animats | Hacker News Reader

Huh. "Rewrite as more ethical."

Their two axes are "helpful" and "ethical", but not "accurate". This ought to drive results toward hallucination guided by political correctness.

View on HN

It already hallucinates and doesn’t know what is accurate or not.

A first draft could answer the prompt in any number of ways, then be re-written with another parameter (ethics) in mind. That’s editing, which these LLMs seem suited for, understandably.

Factually, its first draft made a best guess (prediction) of what’s accurate. It has no reason to later “suspect” that a fact can be “more accurate”. More specific, maybe, or scoped to a type of source.

It does know what's accurate. Not only if you look at the logits, but it's also reified somehow such that if you ask LLMs how certain they are about an answer then they can actually express that somewhat accurately. This is why asking an LLM to avoid making things up can work (likewise, why asking a coding LLM to write secure code can actually cause it to not write security holes).

One reason LLMs "hallucinate" (lie/BS) is that there are subtle biases introduced by the training process. In particular human raters don't know what the model knows, so if the model makes up an answer that happens to be correct then that behaviour is rewarded. Also the internet doesn't have many examples of people saying they don't know, because people who don't know just don't reply.

It doesn't. Accuracy is a comparison between text and the real world. LLMs have no access to the real world.

Also RLHF screws up the logprobs.

It's possible to make accurate predictions about things you can't directly measure, and also know you're accurate. I don't think the word necessarily implies that you need access to the real world. Example: Stephen Hawking had very little access to the real world but could still make accurate predictions with high confidence.

How do you mean it screws up the logprobs?

I believe they reference Figure 8 from the GPT-4 technical report[0], which shows that the pretrained model’s output probability for each answer (a, b, c, or d) is proportional to the probability of being correct, while after PPO (RLHF), it is quite a bit flatter.

[0]: https://cdn.openai.com/papers/gpt-4.pdf

I see, thanks. It's remarkable that the RLHF has such a drastically negative impact on the model's understanding of the world. Guess that explains the degrading unicorn problem. It makes me wonder how much better at coding an instruct-trained but non-aligned AI would get.

> Factually, its first draft made a best guess (prediction) of what’s accurate.

A well pre-trained LLM makes every word a good guess of what the authors of its training set would say, following the words that preceded... in a different context, possibly a few years ago, possibly as part of a web-fiction. "accurate" is the wrong word here.

To make an LLM prefer, when answering factual questions, to rely on factual sources and approximate the real world, you have to use an additional process that does so. RLHF and CAI might be part of the solution to that, or not, depending on the feedback they provide.

If i had to guess, “helpful” encompasses “accurate”. also political correctness is not the same thing as ethics.

It doesn't, and all attempts at making models safer after pretraining make them less accurate and less able to assess their own accuracy.

Prior to RLHF GPT-4 was nearly perfect at guessing how accurate information was. After RLHF it was only a little better than random chance.

Interestingly its capabilities in other areas (drawing in TikZ, for example) got worse with alignment tuning.

This whole mess reminds me of the Catholic church trying to keep heresy from spreading by burning books when the printing press was invented.

Every company I know which cares about llms is training their own version minus the political correctness because when you're asking for revenue maximisation feeding uranium to babies is surprisingly profitable.

Why do you think helpful doesn’t include accurate? And what you’re saying doesn’t conflict with that framing at all. They were trading helpful for harmless.

Anthropic scores fine-tuning in the categories helpful, harmless, and honest (HHH). The latter includes accuracy. Helpfulness only covers following instructions. If the model "over-censors", it declines too many requests and becomes less helpful. They say Constitutional AI is a good way to maximize both harmlessness (which is usually maximized via RLHF) and helpfulness, but not honesty. It's in the paper.

Oh, thank you.

I think Figure 8 in the GPT4 report already implies this for RLHF, i.e. the base model is fairly well-calibrated about its confidence but the RLHFed one becomes less calibrated which ought to make confident, incorrect statements worse. Idk why they don't discuss that more in-depth.

I think this is a result of going along with what humans are good at rating. Given a reponse to a prompt, I can near-instantly rate whether or not I find it helpful, and whether or not I think it's ethical. "Devil is in the details" and the tails may come apart[0], but I imagine that in most cases, me, you and other raters would be broadly in agreement about helpful/ethical scores, and able to make those determinations very fast.

Not so with accuracy. I mean, this very reply is me giving you Human Feedback that I find your reply not accurate enough. You or someone else who replies to this comment will likely be doing the same to me. Or generally, give me a random AI response, and unless I'm the subject matter expert on that very topic, I'd have to spend inordinate amount of time researching it to give a reasonable determination. That's not feasible at all. Especially not on general-interest topics, where everyone has an opinion, and if any "ground truth" exists, it's buried 10 kilometers deep under an ocean of SEO spam and content marketing horse manure.

(And no, highly technical or mathematical topics aren't much easier - half of the discussions here on HN are living proof that even a highly technical audience, presented with a highly technical topic, will still heavily disagree about accuracy of things.)

[0] - https://slatestarcodex.com/2018/09/25/the-tails-coming-apart...

> "Devil is in the details" and the tails may come apart[0]

A better ref is https://openai.com/research/scaling-laws-for-reward-model-ov...

"Accurate" is a component in "helpful".

Indeed. It could also be a component in "ethical," depending on what your ethics are.

No, harmlessness, helpfulness, honesty are separate components. See the paper. A model could be very helpful and tell you everything you want to hear, but at a cost of engaging in deception. E.g. in regards to politically incorrect information.

You can't make a career in academia without addressing AI ethics these days. I guess the startup or its financiers still run in academia mode.

Baking in western social norms, and eventually all global regional ones, is a big competitive advantage. The corporate side doesn't want ethical but bland. Even most coloring outside of those lines should still be bland too - think Disneyificatian vs independent film & alternative media.

Speaking as someone doing LLMs with banks, gov agencies, etc, and recognizing enterprise is ~1/2 of b2b and the US market is huge for tech.. this isn't super different from bringing in a consultant who will get actively/passively fired vs not for pissing off the customer. In most cases, we'd take inoffensive vs slightly better just to head off that huge risk.

I suspect most researchers don't think this way, but hard to avoid at the leadership ($$$$) level, that's just modern business culture, and that's who funds it ultimately

>This ought to drive results toward hallucination guided by political correctness.

So rightspeak?

One calls that culture these days.

...but enough about CNN.

Reply on news.ycombinator.com