Exo: Exocompilation for productive programming of hardware accelerators
github.com
github.com
Also, are there any languages in this vein that take more of a declarative (as opposed to imperative) approach?
Like, I hate SYCL, Kokkos, and all the other similar things as much as the next guy, but major companies and orgs support and use those already.
I personally am not convinced.
So, I've done this professionally (written assembler code, and then scheduled it manually to improve performance). Normally you don't need to do that these days, as even mobile CPUs use out-of-order cores which dynamically schedule at runtime.
It's only going to be useful if you're writing code for some machine that doesn't do that (they give examples of TPU etc)
OOO architectures don't reschedule dynamically - that's impossible - they just have multiple instruction buffers that can issue the instructions. So scheduling is still important for OOO it's just at the level of DDG instead of literally linear order in the binary.
Edit: just want to emphasize
> It's only going to be useful if you're writing code for some machine that doesn't do that
There is no architecture for which instruction scheduling isn't crucial.
In my experience doing back-end compiler work, it's definitely last on the list of major concerns. Obviously you can't ignore it, but it's not where any significant gains are coming from, and anything 'fancy' you do there is likely to be picked up by future generations of hardware.
i have no clue what you're saying here - if your scheduler fucks up your pipelining you're gonna have a bad time (conversely if you're scheduler recognizes pipelining opportunities you're gonna have a good time). as usual anyone who says something like "in my experience it's not a big deal" simply does not have enough experience.
It wasnt until I saw the matmul example that I realized this is (probably, its still unclear) for GPUs.
This made me think of what "accelerators" I've come across (just as a regular consumer):
In the late 90s, that's what we called GPUs - "3D accelerators". For a short time, they were cards separate from the actual graphics card, and often would involve routing your VGA cable through that thing. Before it all merged into one device. I was very slightly disappointed as a kid that I narrowly missed that time, but bilinear filtering and higher resolutions and framerates helped me get over the fact I couldn't cram more cool weird tech into my PCI slots.
Then you had sound cards with "audio accelerators" using technologies like EAX. All that eventually migrated back to the CPU I think.
For a while you could by a "physics accelerator" for PhysX, then acquired by nvidia and moved to the GPU using CUDA. I never had one, but one time I kept around an older GPU after upgrading as a dedicated physx processor. Now that's the only way to run older 32bit games with physx turned up, since that's not supported in 5000 series GPUs.
Finally, I got this "Coral TPU", a USB device with 4GB RAM (and I think around 4 TOPS or something?) for very efficient inferencing. There are some open source projects supporting this, like frigate, which lets you process object detection with multiple surveillance camera streams on a raspberry pi. I never really used it though.
And of course now we have NPUs as sub-systems in CPUs and GPUs. I'd love to have a dedicated PCIe card with that, but of course having yet another computer architecture with dozens/hundreds of GB of redundant RAM is kind of a bummer.
But 93% of the codebase is Python lol. Whatever you think about Python, it is not a systems programming language. Takeaway: this is not a serious compiler project (and it's clearly not, it's a PhD student project).
Deeper take: this is just a DSL that behind the scenes calls a (SMT) solver for tuning (what they call "scheduling"). There are a million examples of this approach for every architecture under the sun. My org is literally building out the same thing right now. Performance is directly a function of how good your model of the architecture is. At such a high-level it's very likely to produce suboptimal results because you have no control over ISA/intrinsic level decisions. Lower-level implementations are much more robustly "powerful".
The problem with this research is that many similar ideas have been tried before in the contexts of supercomputers or CPU compilers. They ultimately all fail because they end up (1) being more work to program in, and (2) not being any faster because real life happens. Networks drop packets, clock frequencies jitter, and all sorts of non-determinism happens when you have large scale. A static scheduler forces you to stall the whole program for any of these faults. All the gain you got by painstakingly optimizing things goes away.
PhD theses, in a best-case scenario, are the basis for new applications. Most of them amount to nothing. This one belongs on that pile. The sad part about that is that it isn't the student's fault that the professor sent them down a research direction that is guaranteed to amount to nothing of use. This one is on the professor.
So I'm not sure you can dismiss it as a thesis which will amount to nothing on the basis that static scheduling is a bad idea!
I could easily have missed something though. It's not a particularly clear or succinct write-up and I have only read some of it. If it does say that it only works for strictly deterministic in-order architectures somewhere please can you point out where?
https://github.com/exo-lang/exo/blob/main/examples/avx2_matm...
The hardware still does what it does, but this is a tool for manually ordering operations.
I did, in fact, read the materials before dismissing them as useless, since this is very relevant to what I do professionally.
I'm confused because that approach seems identical to all mainstream compilers I know of. GCC/Clang also schedule instructions statically and the right schedule will improve performance. Why won't it work here? What kind of dynamic scheduling do you think it needs in order to be useful? Like a tracing JIT? Or they need to implement the reordering in hardware and reschedule as instructions execute?
The selling point of most of these tools is that you can be smarter than the compiler, and this one (despite the example) is selling static scheduling of compute kernels, too. That would normally be up to the OS/driver with some flexibility.
In terms of scheduling compute kernels, the project doesn't remove the ability of the OS/driver to schedule execution. It's only affecting the instruction sequence within the kernel, which is something OS/drivers don't typically control. They retain full control over when the kernel is executed by the hardware.
(PTX is sometimes compiled to SAAS by the Nvidia driver rather than ahead of time. That does allow the driver to reschedule instructions, but using this project doesn't prevent it from doing that. The driver will compile PTX emitted from this project in the same way as any other)
The "novel" idea here is the macro-optimization, with some overtures about making the micro-optimization easier - as you likely know, this is not true since the complexity here is more about understanding how to do better, not about what language features you use to make those changes.