Hacker News Reader

Comment by bottlepalm | Hacker News Reader

I think a lot of the upvotes, including mine, are just realizing that this is a thing. We can see implications of this as the model sizes scale up, just like early LLMs weren't as impressive, but we could see where they were heading.

View on HN

Scale is an interesting point here. Large overparametrized networks don't behave in the same way as small toy models do; this is one of the big caveats of many interp works. I expect naked weight edit will not work for large models. Why? Because an agent needs to know what it edits and reason on it. You could theoretically improve a model on some form of weight recognition; maybe it will work, maybe it won't. But again, we come up against a cost/benefit dilemma; it sounds exotic and sci-fi but likely impractical. Parameters of the model are connected and relate to each other. Instead, current SOTA auto-interp argues that what we need are better tools for models to know themselves, so rather than direct edit, we need to give models better understanding of large parameter geometries, then they apply edits with non-destructive approaches at scale. This is much harder to do than applying a steering vector.

> I expect naked weight edit will not work for large models. Why? Because an agent needs to know what it edits and reason on it.

Would that still be true for MoE models where only a small part is activated on a specific topic?

I think _small_ is still doing a lot of work here. Yes, an MoE model doesn't "use" all of it's weights at once _but_ what it does use is still a hell of a lot heavier than the 20K toy shown in the paper.

I agree, I’m impressed by a sole idea of connecting these two things together

I upvoted without looking into the details because it sounds cool, whether it's new or not.

I suspect a lot of upvotes also come from people afraid of closed western labs running away with AI. Similar to other articles with positive news about open weight models.

RL the model on weight editing

The #1 thing frontier AIs are optimized to do is create/train new AIs.

Reply on news.ycombinator.com