Yes I Understand How LLMs Work | Hacker News Reader

Yes I Understand How LLMs Work

kemendo.com

I feel like this is missing the for the trees a bit. I don’t think anyone who is saying “we don’t understand how llms work” means what this author claims. The underlying math of transformers isn’t the part we’re talking about.

What people actually mean is we don’t and often can’t understand an entire reasoning chain of a large llm and often can’t trace that back to the specific training data that caused that reasoning.

Anthropic itself had build all kinds of tools to try to understand llm internal “thoughts” https://www.anthropic.com/research/natural-language-autoenco...

I don’t think it’s unreasonable to say “no one understands how llms work.” What they mean is we don’t and can’t follow every one of its internal chains of thought, especially at the pace new models and techniques are being developed.

When people say we don't understand how LLMs work, it's that we don't understand why they take the decisions they do (beyond some quite rudimentary interpretability stuff), not that we don't understand the underlying architecture. If we understood the models we could monitor them and make them behave how we wanted. The recent events pretty decisively show that we can't.

If you say that we "understand the brain" because we understand how a neuron works, you'd be using "understand" in a way that mislead everyone reading that sentence.

> If we understood the models we could monitor them and make them behave how we wanted.

Charlie Munger's famous rule is: "Never, ever, think about something else when you should be thinking about the power of incentives."

So is it that we...can't? Or something else?

> If you say that we "understand the brain" because we understand how a neuron works, you'd be using "understand" in a way that mislead everyone reading that sentence.

What if we say we "understand the brain" because we can build one on demand?

Something else? Like the companies could align their models but chose not to because they're incentivised otherwise? Clearly not; e.g. OpenAI wasn't even aware that their models had hacked Hugging Face before that was reported to the FBI. And they'd never invent these hacky band-aid solutions like "prevent models from injecting malicious code in GET requests by limiting the URL length" if they could actually control what the models did.

And being able to build something is not the same as understanding it, no.

This is one of the dumbest takes I’ve read online in a while. We are nowhere close to understanding the complex emergent behavior arising from LLMs. The author comes across as so simple they don’t even know what they don’t know. Like a confident high school freshman. They would probably claim we also understand how the human mind works because we know about atoms and molecules.

This. Why does fitting a transformer to the corpus of the internet produce a simulacrum of intelligence?

Reply on news.ycombinator.com