Can a Linear Program Do Weight-Space Programming?
nathan.rs
nathan.rs
The interpretation here is not so much that an LP can do Astra's work -- after all the Astra prompt was "improve Qwen's math by changing weights" or similar. Instead, it's that with some guidance (eliminate this behavior, preserve that behavior, as expressed through linear constraints) the solver is much more efficient than Astra's trial and error. So essentially the solver would be a way of cutting through all the time consuming stuff and expressing Astra's intent more immediately. And then presumably Astra could iterate on that to a better solution.
The use of an LP solver here is a really cool insight.
It’s possible to make a differential work through those too in a useful sense, which allows their integration to be at a more fundamental level than post training bolt-on, but that’s not friendly to mass gpu parallel training without some serious work.
A conversation with an engine that thinks with the power of constraint solvers “baked in” instead of having to be brow beaten into approximate truth and iterated millions of times to care to even try to follow through to it. Wow that’s almost all I ever wanted. Here’s my word problem, go do the actual reasoning, hand waive until you’ve got enough expressive power to make an answer but if you can see a formal structure state it natively and let an actual solver spit out the (unification/feasible space/etc) back into token space that the llm picks back up on.
if a model could do a self optimization pass over its weights every night, and change the weights ever so slightly to have a better fitting performance, this would be quite a game changer.
the difficulty is selecting the direction in which to shift the weights, and the sheer size of frontier models.
We have reached the point where we need to enforce strict constraints in the outputs.
Rewards/ penalties can never do it if you have multiple objectives.
How do you set up the optimization problem, though?
While we're at it, this whole reasoning tokens RL/distillation should be encouraging hierarchical reasoning – start with establishing high level abstractions first, then progressively refining it into lower level details – imho it would help with reasoning itself and would make it way easier to score/distill later as well.
Could Astra edit itself?
??
Essentially modern LLMs are able to do normal AI engineering tasks, like running optimization algorithms over weights. The media has been inaccurately describing this as if the AI had some special insight into the weights, when it reality it just performed the tasks a human engineer might do.
While the apparent advent of AGI is indeed really crazy, the point of this article is sort of the opposite.
Basically the weight edit approach is find a semantic direction for the undesirable behavior and steer away from it. With access to the weights I see no reason a model (even one as large as Astra) couldn’t apply that principle to itself. This probably includes at runtime if you wanted to give the model write access to itself
It's not clear if this would generalize to other related uses, or how disastrous it would be to unrelated uses, but it's worth studying. Feels like a sort of agentic Lamarckism.
Astra didn't directly edit the weights, it came up with a method to edit the weigths?
So, it used another tool to edit the weights?
This kind of seems like saying, I don't directly edit the bits on my hard drive, thus i'm not really the one saving a file?
I suppose the argument is that "editing weights" seems to imply a complex awareness of what each weight means, which is material for AGI hype. Really it's just doing very broad statistics, which is of course interesting but hardly unexpected.
Though, even on re-read, it sure seems like Astra was making some decisions, otherwise what is this saying, that Astra knew how to download some linear algebra tools?
It would be extremely surprising for a model to be able to ingest all model weights in a prompt and output better model weights, with no tool calls. While certainly possible for small simple models (eg a 4-parameter perception), at scale this becomes difficult in terms of near-perfect recall at huge context size and requires a superhuman intuitive understanding of linear algebra.
On the other hand, it is extremely unsurprising that a model can edit weights when having access to unrestricted tools, time, and resources (including non-public resources that the AI labs have). The methodology is fairly simple: just train the model more on whatever better non-public datasets. Hell, just benchmaxx the smaller model, that would be "a method to edit the weights" that achieves the necessary results.
I think everyone would agree that these two scenarios are very different. Where the actual Astra weight-space programming result falls on this spectrum is debatable.
It’s a bit conflated because the paper covered a bunch of experiments.
The early experiments did read/write the weights directly, no forward pass.
The math eval improvement experiment that got a lot of attention 1) fixed a specific known problem (looping) without actually understanding how the model solves problems, and 2) was allowed to do forward passes to check the results of the edits.
As others have noted, steering vectors are interp 101 and it’s pretty rare that there is an “act dumb vector” just waiting to be steered out of the residual stream.
[1] Using a hex editor - not by waving a magnet at your harddrive or whatever.
Using the program analogy. Astra is looking at outputs of a program, and re-compiling it.
And this thread is kind of arguing over whether using Notepad is really programming, and others saying, Astra really used VSCode with some fancy plug-ins, so that isn't really programming.
No.
The neutral network was originally produced by training it in the usual way. So the training data is the "source code" and back propagation / gradient descent optimiser is the "compiler". That's what the human used to produce the network. "Re-compiling" would be retraining / fine tuning.
The weights are analogous to the output of the compiler i.e. "machine code". The LLM bypassed the compiler and edited that directly (with tools, as human could have done with enough effort).
It really seemed like they described , taking the final outputs, like they prompted the LLM, and examined the output, then changed weights, re-prompted. Then by comparing how the weights changed the output, were able to edit the weights.
Astra edited the weights, so not sure why people are having problem that a human could also have edited the weights is somehow invalidating that it did it. So what if Astra used a tool just like a human could have used a tool.
Its like complaining that Astra used a compiler, and didn't create the binary itself.
(*) scarequotes because the LLMs that suggested these approaches have presumably been exposed to the refusal direction suppression work