Sorry to be cynical, a lot of smart people seem to have built this.. but I'm failing to see how it's any better than "hey claude, here's my program and a profiling tool, go do your stuff"
(a) Scope: Maxwell can optimise portions of both your own code, but also of underlying libraries too (think libc), according to your data flow and env. These aren't places where you necessarily have source code or an easy way to replace it.
(b) It's dynamic: like a JIT, if your data flow _changes_, Maxwell detects new bottlenecks and reoptimises. A good example is an inference pipeline where e.g. the concurrency or seqlens can change suddenly (depending on what your users do). If you reoptimise for each such case, you'll do so forever (and there are many different operational points, each requiring their own algorithmic approach).
You say "think libc" - are you suggesting they could dynamically recompile bits of libc and do better than LLVM's link time optimiser already did it??
What I assumed was that you'd run this thing on some test runs, as a profiler step, and create an optimised binary. But you say they expect to run your code through an LLM all the time!?
The idea that you could run your code through an LLM (which must surely slow it down 100x at least), and then find enough missed optimisation opportunities to make up for that slowdown is nuts! (Or I've just misunderstood the whole premise!)
Honestly, the places in programs where you can get meaningful performance gain (that isn't already handled by your compiler) is all at the high-level big-O level, and that doesn't require runtime inspection.
The binary is running. Maxwell is watching it: presumably there is some cost to the watching, but the LLM is not running in the binary process. Maxwell also has dynamic information about the running environment. Maxwell can then try and superoptimize the running binary using the dynamic information it has, but to do so, it needs to create semantics-preserving patches to the running binary.
What should be clear here: dynamic information is strictly more information than what a static AOT compiler has access to.
So that's the problem: we're going to patch the binary while it is running, and we need the patches to preserve the semantics as expressed by the high-level program, possibly assuming that the compiler was semantics preserving, etc. I'm assuming they have custom compilers or something for all of this.
Getting this to work probably involves creating some sort of source map, perhaps a custom linker, etc (just guessing) from high-level program to binary -- but assuming you handle all of that, then the job of Maxwell is just the job of a normal superoptimizer: propose semantics-preserving patches, see how they do, etc.
In theory, you could propose terrible patches frequently - superoptimization normally discards these, but you with bad proposals you might spend a long time searching for good patches. But LLMs are good proposers now, so you'd probably mostly get good patches (to a limit).
Authors, correct me if I'm wrong.
But if:
- we're talking about compiled code, where the compiler has already squeezed out all the low hanging fruit in terms of performance
- it's running some kind of external process somehow watching the data flow around in memory, and is figuring out what code is doing what. This is at best incredibly expensive, even if happening in parallel
..then the chances of actually finding useful micro-optimisations that are also safe to patch while running without changing the memory layout and messing up.. is surely pretty close to zero.
I can only assume that it's not actually native code, and they've built some kind of python JIT engine which is finding efficient replacements for chunks of common python, where it starts off so slow that even the massive amount of effort required manages to pay off in a long enough run.