People haven't spent time optimizing the fp32 versions of these kernels in years. This will be much more interesting if they can improve the kernels where developer effort has gone and that are actually used.
For a processor with well-documented microarchitecture, for which a programmer or a compiler can deterministically write an optimal program, it is much less likely that applying ML/AI can be successful, except as a substitute for searching already known solutions.
On the other hand, for less documented microarchitectures, like of the NVIDIA GPUs, finding an optimal program may be impossible other than by doing a random search guided by examples of previous optimized programs, and possibly doing some reverse-engineering work to determine the real behavior of the GPU in some circumstances.
Improving over something like this is likely to be feasible for ML/AI, where training over known good programs may be able to extract some of the undocumented behavior that may be non-obvious for humans reading those examples.
For register allocation and instruction selection, there is hope because it is FPT and there are algorithms to do it optimally in polynomial time, albeit with a large constant factor (FPT), making it impractical to apply to compilers as of today. For instruction scheduling, it is just too hard. If you read literature on scheduling algorithms, it is NP-hard for apparently simple instances, e.g., 2 parallel identical machines with no preemption and bounding completion time (https://www2.informatik.uni-osnabrueck.de/knust/class/), while actual microarchitecture is much more complicated than this...
Needless to say, these are already the simpler problems. The longer the program or the more profiling data you can optimize for, the more tricks you can throw at it, and most of them are NP-hard to optimize optimally.
Being NP-hard doesn't imply that you can't obtain the optimal result, but compilers that I know of do not implement them, because most users are not willing to wait for days for such a compilation to complete. Ideally, one should make something that can run on clusters of CPUs or GPUs to optimize this, and people having those clusters will typically be willing to do this because they want to optimize the program they later run on the clusters. However, to my knowledge, no one is working on this at the moment.
The latter happens when there is one dominant bottleneck for the algorithm, which is determined by the hardware, e.g. the maximum throughput of a certain instruction, like multiplication or memory load. When the implemented program reaches a throughput almost equal with that absolute limit, then one can be certain of its optimality.
You severely underestimate the landscape of possible implementations for these kernels. There are many ways of performing a matrix multiplication and predicting which one will perform best without running them all is nontrivial, even with perfect knowledge of the underlying system.
This is just a completely incorrect take, speaking as a former insider.
For a processor that is completely documented one must be able to run a simulation model that provides the running time for a given program, when the processor is so complex that simpler methods for computing the execution time do not work.
For older NVIDIA GPUs, there exist such simulation models, but they are only partially accurate, because they are based on reverse engineering, without cooperation from the GPU vendor.
You can't precompute all the possible options in advance and fetch the running time from a database either, because the parameter space is just way too huge.
Notice that none of this has anything to do with having accurate models of the system. This is what people who do this for a living and have perfect knowledge of the system choose to do, for good reasons.
We don't even know the optimal algorithms! AlphaEvolve recently found "an algorithm to multiply 4x4 complex-valued matrices using 48 scalar multiplications, improving upon Strassen’s 1969 algorithm that was previously known as the best in this setting." - https://www.nature.com/articles/s41586-022-05172-4
Wow, so, you're basically saying the AI created new algos in a domain with no pre-existing solutions? Awesome!
The implication was that the FP32 versions of these kernels have lagged behind the more popular versions. There was opportunity to translate the advancements from other kernels into these. Someone would need to look closely to see exactly what was done, but it’s premature to suggest anything like “new algos” or “no pre-existing solutions”
This is a great use case for LLMs, though. I often do something similar where I make improvements to something I use most frequently and ask an LLM to translate that pattern to other similar parts of the code.
Help me understand this 'cause I'm a bit slow these days ...
Does that mean optimized FP32 versions of these kernels were already there or not?
If I do `sed 's/f32/f16/g' kernel.cu` does this count as AI? Help me understand because I'm a little slow when it comes to all the dumb shit people attribute to LLMs these days...
>sed 's/f32/f16/g' kernel.cu
This is not what's happening here, it's a completely different thing, read TFA.
If you're trying to support your original point with that argument, then you're using some pretty awful definitions of the terms "new algos" and "no pre-existing solutions".
That being said, it is cool if AI is enabling lower cost adoption of better more optimized kernels with less effort.
Read the damn comment you're responding to. There have been human written kernels for both fp16 and fp32 for a long time.
Here is the corrected version of your comment:
"Wow, so, you're basically saying the AI created the same but faster algos in a well known domain with established pre-existing solutions, whose overall impact on the runtime of practical workloads is insignificant? Awesome!"