Hacker News Reader

Comment by HAL3000 | Hacker News Reader

This implies that weights are a type of language. Neural networks are good at language because they are good at patterns. They can reason about it because their "reasoning" isn’t how we think. The same goes for how they encode images, their vision is entirely different from ours. That's why adversarial attacks work on them and not on us. We don't see anything change in the image, while the neural network now sees that the dog is a cat.

View on HN

Ok? But does that mean that the “weights language” is universal? E.g. same weight patterns mean the same thing regardless of if it’s Kimi, Grok, or Astra?

There could be different patterns that solve the same problem. Just as you can have different algorithms, there could be different weights, but the abstracted result or logic from those weights would be the same. Take these words I'm writing now, they will be transformed into numbers, then sent to your screen, and then your brain will transform them into analog signals. I bet the neural networks in our brains have "different numbers", but the algorithms that understand language and other concepts are similar and could be universal.

It's like a sorting algorithm, which I can implement in many different ways in just a few dozen lines. If you look at the binary code (so like raw weights) of the compiled program, you will see just a bunch of a different numbers, but if you read the code, in most cases you will understand that it's a sorting algorithm. So it seems they just operate on different levels of abstraction that are not available to our brains.

i think a better question and one that probably has a lnown answer is whether there are infinite nontrivial solutions to train a neural network of a certain size to match a function? Trivial here would be about simplifying `n * 2 / 2` from two neurons to just passing n from the input

do they break down to a finite subset of groups?

How do we (or can we) know adversarial attacks don't work on us?

We're pretty confident that adversarial attacks do work on us, actually—we have entire industries built around it. Some of them even start with "adver-"...

There are classes of adversarial attacks targeting them that don't work as well on us, and vice versa.

That's not the point I'm making. So in context of vision we pattern match too, and you can see it in the hundreds of optical illusions online. Our visual system relies on built in shortcuts, and they break in different places than a neural network's do. Image models get fooled by things we would never notice, and we get fooled by things they handle fine. IMO that's good evidence that we and these models use different algorithms to solve similar problems. I think the same is probably true of LLM "reasoning", there's some overlap with how we think, but a lot of it likely works very differently under the hood. Good example of this is anthropic's interpretability research where they found claude doing addition in a way that's nothing like the carry the one method it describes when you ask how it did it.

Hypnosis and optical illusions are adversarial attacks on shortcuts our brain takes.

As StilesCrisis said, adversarial attacks do work on us. (naniwaduni makes a very good point, as well.)

Highly relevant: "BLIT" <https://en.wikipedia.org/wiki/BLIT_(short_story)>

Also, <https://www.quantamagazine.org/a-math-theory-for-why-people-...>

SCP calls such things cognitohazards <https://en.wiktionary.org/wiki/cognitohazard>.

Reply on news.ycombinator.com