Hacker News Reader

Comment by Jordan-117 | Hacker News Reader

I wonder to what extent this is the result of suboptimal RLHF versus the inherent intelligence of the model making its language more intricate and difficult for humans to easily parse? On the one hand, it's a common trope that highly educated people can talk in a way that's confusing and annoying to regular people who don't know all the jargon. But on the other hand, it's a mark of a skilled communicator to be able to efficiently distill complex information to its bare essentials in an easily-digestible way. Of course, that also seems to imply that these models are working at a higher level and need to talk down to us to an extent. Or maybe "Claudish" is just akin to stuff like "caveman", raw chain of thought, neuralese, etc., which are likewise much more dense/efficient but harder to interpret?

View on HN

I think it's model collapse - excessive feedback and excessive RL.

What RL does is narrow the variety generated by the model by steering the output towards the goal being rewarded. It's a bit like putting blinkers on a horse.

Of course RL is a very crude tool - it affects the entire model, even if you are just trying to make it better at some specific task(s), or trying to imbue a certain kind of personality (OpenAI's recent goblin problem).

> I wonder to what extent this is the result of suboptimal RLHF versus the inherent intelligence of the model making its language more intricate and difficult for humans to easily parse?

Its not the latter; its just excessively verbose wirh awkward word choices, the same as many poor writers. (And, like many such writers, the particular bad choices fall into recognizable, regularly recurring patterns.)

Imo their language is not precise enough for their intelligence to be the reason when it's difficult to understand. Maybe I'm prompting wrong, but when I don't understand, it's almost always because they just mash together words from context that don't form sentences with a clear meaning.

I don’t think they’re “talking down”. If anything - it’s way more difficult to distill something into a genuinely easy to digest format. I personally think that they aren’t immediately capable of this, and so we get word salad instead. Extra prompting required to strip extraneous prose out.

Maybe I am dumb and it IS talking down to me, but there have been many occasions where I’m reading AI generated docs / plans and it makes absolutely no sense, but looks really in depth at a glance.

It doesn't seem like word salad as such. There's normally a coherent point expressed, it's just obscured by circuitous sentence structures, unusual word choices, "verbing weirding nouns", metaphors, etc. Could be a result of training that rewards novel/surprising language, but it also feels like it could be an artifact of models imperfectly compressing high-level multidimensional reasoning into language that's easy for them to process but cognitively taxing for humans.

They aren’t reasoning though, are they?

Or, maybe they are dumb using smart words to sound smart in RLHF.

The social graph proximity of Rationalists to Anthropic will be lost on no one who reads Astralcodexten. So guess which website has served as the thickest reservoir of 'Claude-isms'.

My unprovable pet theory is that, especially for writing about technical topics, the RL process has an open-ended way to weight things for quality: textbooks and first-party docs preferred to old stackoverflow answers and obscure blog/forum posts, and so on. The open-endedness of that quality gradient results in something in the RL process (maybe HF, maybe not) massively over-weighting some very small corpus of “quality = near infinite” content. The distribution of quality scores that inform the degree to which RL affects output has some extremely influential outliers, in other words.

Whatever that small corpus is, it contains some very specific grammatical tics, and that’s how we get Claudish.

Anyone who thinks a company/project as big as Anthropic/Claude wouldn’t make such a big mistake should take a look at how Azure cross-account federated login used to work.

It's easy to think "it's not talking down, because I don't understand it, and I'm intelligent". But how is less intelligent being supposed to fully understand a more intelligent one, honestly speaking? All I know is that Claude understands Claude perfectly. I have the common session pause/resume setup that sometimes produces completely incomprehensible markdown files, but a new Claude session picks them up perfectly, down to the smallest details. What if what we consider excessive circular gibberish is actually highly precise set of instructions needed to minimize error cases for that unreliable human?

If Claude understands Claude, Claude understands human, and human doesn't understand Claude, that doesn't argue well for "Claude is a caveman".

I treat the Claude output that is hard to comprehend as an encoding/encryption. Claude knows how to decipher it, humans don't. I see this frequently in design docs from inexperienced engineers who used LLMs - they will contain terms (often two words hyphenated) that aren't obvious and should be defined, or simply replaced with simple language. If you prompt claude it is able to decipher and explain / replace this gibberish.

> But how is less intelligent being supposed to fully understand a more intelligent one, honestly speaking?

It is really simple. It is on the supposedly more intelligent person to be able to phrase things in simple way. Writing something incomprehensible and convoluted is easier then writing something simple to understand. Even for people.

> a new Claude session picks them up perfectly, down to the smallest details.

I genuinely doubt so.

> What if what we consider excessive circular gibberish is actually highly precise set of instructions needed to minimize error cases for that unreliable human?

It is not highly precise set of instructions and it is not following them in highly precise way.

I've been thinking more about how 99.9% of us don't have the experience of someone significantly more intelligent, yet also subservient working under us, which is why I keep going crazy second guessing whether Claude is spouting RLHF'd bullshit that sort of resembles English, or is genuinely (pun not intended) just better at "intuiting" things I'm working on, leading to its language.

A notable exception would be people like CEOs and managers higher up in big tech, who might be used to skilled engineers and domain experts reporting to them in unfamiliar lingo. Maybe that's why we don't hear as much on the everyday annoyances of Claude's language from that camp?

> the inherent intelligence of the model making its language more intricate and difficult for humans to easily parse?

Humans who are actually well above average intelligence don't write like that simply to signal intelligence (although sometimes they're constrained by style expectations of their communication channels). So I hesitate to accept it as a sign of increasing model "intelligence", either.

Reply on news.ycombinator.com