Hacker News Reader

Comment by youareundersell | Hacker News Reader

I might be mistaken but I have the feeling you are not getting the point of this article. Your example is about identifying "facts" in a network. While very interesting is not what strikes me as the main point. The raw ability to state things about NNs is indeed an existing field, but...

The fact you can give an LLM a raw dump of neural weights and it will hand-wire a bunch of changes on the fly is quite something. It's not so much about the OMG! RSI! for me personally, it's more about the fact we are dealing with a system that reasons successfully over a raw bulk of extraordinarily opaque data without breaking a sweat. That ability is alien and we need time to adjust to that.

View on HN

> about the fact we are dealing with a system that reasons successfully over a raw bulk of extraordinarily opaque data without breaking a sweat. That ability is alien

But it's not what's happening here at all! Read the thing in detail and you'll see that it's not doing anything alien, it does exactly what you (or at least someone with at least some knowledge abouy neural nets) would do if you were asked to perform the task. It doesn't reason about weights in a magical way, instead it reads the easy to interpret parts or use directional ablation to tweak a model's behavior, exactly like how the ERP folks have been decensoring models to produce adult content over the past two years.

It's an interesting demonstration of ML knowledge and agentic ability, but it's not alien NY any means, and it's not actually surprising given how strong today's models are with agentic work.

I think you are over-indexing on the word alien. What matters here is how scalable the technique is and the fact that it is autonomous. This can be a breakthrough for continual learning if it can be trained to work at scale.

I think this other submission from yesterday that visualised the examples was quite helpful:

https://news.ycombinator.com/item?id=50344861

Thank you for sharing that, it's a shame that the overhyped paper received this much attention while this one ended up burrowed beyond eyesight.

Thanks for sharing. I was also onboard the hype train ready to share this widely and get others onboard, but now I'll be sharing this article instead.

I read the report; it's not complicated. I'm sure there is some basic misunderstanding here.

When I say "alien" I don't mean "magic" and I don't mean "emergent new woo-woo scary ability". I mean, not human in speed/volume. No human can process that amount of raw bulk in that amount of time successfully. And indeed, it's not magic at all. It is enormously mundane actually. It reminds me of old computers being able to do arithmetic at insane speeds compared to humans. It's not magic at all, but to be honest, I have the distinct feeling of goalposts flying all around me. Being able to handle tasks of this complexity.. I don't know about you but I still have to adjust to computers being able to do this and that it's actually "not surprising". I guess I'm getting old.

I can see how you want to calibrate the enthusiasm I see around here as it can be a bit.. well, but it's not necessary for me. I'm not overly enthused by this because I don't care about the NN adjusting itself and RSI etc. I'm sure it's all quite basic but the fact it can do it at all is reason to be.. watchful, IMO. That is to say, I can appreciate the level of capability necessary to do this work successfully, but I do take your point seriously. It made me think, thanks.

I wonder to what extent this ability exists because Astra is used/trained for internal development purposes at OpenAI, and to what extent asking it to improve weights on other models leaks internal knowledge.

It would be interesting if asking astra to edit your model becomes a new way, like distillation, to train your model on the cheap by extracting OAI's model work.

HN needs a mega-downvote button

Reply on news.ycombinator.com