That doesn't mean I don't like progress mind :) This morning I got the preliminary results of a 'C' front-end to the compiler (it uses clang to compile to LLVM IR, then translates to our IR, then feeds that IR into the autovectoriser as if it had parsed the code itself). So for the exact same source code I now have:
┌─────────────┬────────────┬──────────────────────┬──────────┐
│ Benchmark │ clang -O2 │ xcc --par-auto (GPU) │ Speed-up │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ saxpy │ 1,546 µs │ 726 µs │ 2.1× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ Mandelbrot │ 394,238 µs │ 2,110 µs │ 187× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ nbody │ 37,090 µs │ 1,035 µs │ 36× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ perlin │ 78,016 µs │ 1,850 µs │ 42× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ mandel_deep │ 73.2 s │ 55.8 ms │ 1,312× │
└─────────────┴────────────┴──────────────────────┴──────────┘
Hacker News isn't interested - any post I make linking to the webpages is immediately blocked/dead so I'll wait until the C++ front-end is ready (some time next week) and announce it somewhere else. A compiler that takes standard C/C++ code and speeds it up by automatically finding data-parallel areas and scheduling them on the GPU with no programming overhead is pretty rare.
Oh, and it manages the local network as well, so you can run xcd on machines on your lan, or via the TB4/5 bridge between Macs and it'll schedule on other GPUs, just like it manages multiple GPUs on the local machine. It characterises both latency and compute-time to figure out which devices are worth scheduling on.
Bottom line: GPL3 for the compiler, standard library exclusions for derived works, so use it for anything but feed me back improvements to the compiler itself, and you don’t get to redistribute it as your own.
Still working on some of the data-parallel recognition in LLVM IR space. In order to support both C and C++ with the same data-parallel section extraction, I moved the detection from the ‘xc source code’ phase to the IR lowering phase, meaning that everything could ‘feed in’ to the same routines. That’s caused a little friction with the slightly different ways the LLVM IR from clang and xc’s IR encode loops. The above results are all single-loop extractions and the double-loop ones are still ongoing.
The system works in 3 stages: 1) by being greedy about which loops could possibly vectorise and marking them, 2) by analysing the IR to reject loops which do pointer-aliasing, or object manipulation, or… and then 3) by measuring the fastest way these known-safe kernels could execute.
Which brings me onto one nice thing I haven’t mentioned: that the GPU choice(s) are made at runtime, not compile time, and the system automatically measures (and then persists so it doesn’t have to measure again) the fastest way to run a kernel on the current machine. It handles MSL on Apple devices, PTX on Nvidia cards, Spir-v on Windows, Linux, and Android, and WGSL in the browser. It’ll quite happily divide up the parallel task across a 3090 and a 5080 on my Windows test box for example, with the 5080 doing ~2/3 of the work, because the scheduler is opportunistic.
All without writing anything other than plain C with nary a mention of the GPU. Give me a week or two, and I’ll probably announce it on phoronix (which might then get a mention on HN if it goes down well there, and it’ll be a different url so hopefully won’t be insta-dead).
I wouldn't be too sure about that :)
I think it’s my own fault, I’ve posted the link to the compiler pages several times in what I thought were relevant posts, and I think HN thinks I’m spamming. So I’m not posting the home page for a while and we’ll see if a future submission “takes”
For anyone wondering, the thing lives at https://compile-xc.org/ ;)
The C front-end I'm currently working on is something that looks at the LLVM IR that clang produces and converts it to xc-IR, then runs the data-parallel extraction through the same routines as the original xc language. Consider it to be a 'Y' branch on the inputs - you can feed it LLVM IR or xc source code, they both get convered to xc-IR and from then onwards it's all the same.
So yes, you get tiled work-units, overlapping in time between compute/data-transport and tuned so that actually works, and an opportunistic scheduler that sends them to whatever worker can handle it at the instant of being sent. That opportunistic nature means the faster cards (eg my 5080) will naturally get more work, because the scheduler just keeps all cards busy. The divide between the 5080 and the 3090 in my Windows test box is about 2:1.
But the scheduler is also aware of networked GPUs - my M4 max has a pretty fast GPU, but it's connected to my wife's M3 Max by thunderbolt, and I measured a 37 Gbit/sec connection, so my app can send data to be processed over that very fast link and get results back. If you have a 10GBit nic, then it will work pretty well too, if your nic is "only" 1Gbit then modern GPU speed means that unless your work-units are really computationally intensive, it'll be faster to do it locally - and xcd will work that out at runtime and adopt that faster approach.
- Acquisition: Greedy annotation of any possible data-parallel opportunities. This was originally done at the source-level when it was just 'xc'. For the C front end, it's been moved into the IR level, where the xc version will migrate to soon as well.
- Validation: Always done at the IR level, here we can check that there are no data-hazards, that a variable isn't being read from and written to in the same loop for example, or any aliased pointers, etc. etc. Anything that doesn't meet the cut has its annotation stripped, and we fall back to CPU, though it may still be vectorised on the CPU.
- Runtime measurement of which method is fastest, on the hardware that's actually running the code, with persistence to the local machines settings (defaults on a Mac, registry on Windows,...). If the routine passed muster during validation, it's guaranteed to be tile-able, and splittable over multiple GPUs in the system, or even networked co-operating machines (or Macs over Thunderbolt connections).
So, yes, the workers only see a tile of data, and there can't be a dependency within that tile, but xc allows trivial data-parallel annotation of loops. Here's a full-and-complete Mandelbrot that will run on a GPU:
#use "Stdio.xc"
#use "Par.xc"
#define W 2048
i32 main(void)
{
u32 total = 0;
par mandel :reduce(+ total)
{
for (u32 i in 0..W * W)
{
float cx = (float)(i % W) * (3.0f / W) - 2.0f;
float cy = (float)(i / W) * (3.0f / W) - 1.5f;
float x = 0.0f;
float y = 0.0f;
u32 k = 0;
while (k < 256 && x * x + y * y < 4.0f)
{
float xt = x * x - y * y + cx;
y = 2.0f * x * y + cy;
x = xt;
k = k + 1;
}
total = total + k;
}
}
printf("%u iterations\n", total);
return 0;
}
- Discrete GPU: a GPU card (Vulkan or Cuda) keeps resident any array the program hasn't written since the last upload. Running a block again that uses that same data uploads nothing.
- Remote: a remote xcd keeps every slice and kernel it has been sent, so a repeat run sends hashes, not data. Same story as above for a discrete GPU managed via xcd
- Metal and integrated Vulkan: simd: memory isn't copied at all.
...so if you want something that uses a buffer from a previous data-parallel compute, have 2 'par' sections, and share the buffer. Each 'par' section will still run serially (en-masse) by default, but you ought to be able to ping-pong between a pair of them. I haven't yet run such a thing on the CFE :)
More specifically:
- Within a block, iterations must be independent. A loop where each iteration reads the one before (b[i] = b[i-1] + 3) stays an ordinary loop.
- Reductions are the one cross-tile dependency it handles. Each tile, on whatever device, produces a partial sum, min or max, and the runtime combines them. That's what the :reduce(+ total) does above.
- The scheduler sees the topology, not the workers. It splits a block into tiles and hands them out by each device's measured speed, local or remote. Each worker sees only its tiles plus the slices of input those tiles read.
- For streaming, two blocks in sequence works. Each block finishes on every device before the next statement runs, so the second can safely read the first's output. With CUDA residency, data the CPU hasn't touched in between isn't re-sent.
One thing I'm not trying to do is cover all possible cases automatically - that way lies unbounded complexity, and I'm a fan of simple code working well with low cognitive load. xc will work well automatically for non-self-dependent data, and allow you to work far more easily with self-dependent data.
Thank you!
I was all primed to give an explanation about some of the internals of the language, and after checking ... that's not whats going on here. What I thought was happening (and why I wasn't overly surprised by the saxpy results) was this:
- The above data is actually from my M4 Max MBP, not the PC using a 3090/5080/both
- xc has the concept of memory that is sufficiently aligned that it can be used directly with GPU data in a unified memory model. You just annotate the pointer with 'simd:', thus:
simd:u8* ptr = new u8[4096]
That ... turns out not to be the case [grin]. The CFE is not marking buffers that get allocated on the heap with the simd: annotation (this is a bug, I will fix it), so you are absolutely correct and the numbers as presented above don't stand up for saxpy. Here's a more in-depth table for saxpy, and it shows where the error crept in:
┌───────────────────────────────────┬───────────┐
│ saxpy, M4 Max (C, --par-auto) │ Time │
├───────────────────────────────────┼───────────┤
│ clang -O2, one thread │ 1,398 µs │
├───────────────────────────────────┼───────────┤
│ xcc, all CPU threads │ 518 µs │
├───────────────────────────────────┼───────────┤
│ xcc, auto (picks all CPU threads) │ 593 µs │
├───────────────────────────────────┼───────────┤
│ xcc, GPU │ 11,240 µs │
└───────────────────────────────────┴───────────┘
The 'auto' mode (the default unless you pin a kernel as must-use-this-execution-style) measures a few iterations of the work-tiles at runtime, and picks the fastest to run the full job with on the local hardware at runtime (unless it has metrics from a previous run).
Bottom line: what is actually happening here is that the 'auto' mode (also in the original table header) is doing its measurement, and picking the fastest route - which is the "all CPU threads" mode. You can see the measurement cost (I always run these tests after clearing out the persisted previous run, so there's always the measurement-cost in the 'auto' figure). What was misleading was the "(GPU)" in the original table header, I have to apologise for that.
Having said that, I'm glad you asked - in this case I don't think it'd make an actual difference (multi-threaded CPU still beats the non-buffer-transfer GPU) but potentially an integrated GPU (Mac or one of the "AI" Ryzens perhaps) might benefit from another, very slightly more complex compute, now that I know the automatic memory-alignment isn't working properly :)
Yeah, for an M4 Max I'd expect 1.5-2x (the CPU can only use about half the total bandwidth, roughly, so the GPU should still win if you discount initialization), though if memory serves the Metal API needed some special flag set in addition to the basic alignment requirement in order to actually directly use the same buffer without extra staging, at least when I last used it.
More generally, if your language builds a unified memory abstraction on top of systems that have physically heterogenous memory, make sure the cost model doesn't just include the nominal bandwidth differences (or asymptotic ones, if you do live profiling), but also the overhead of any extra staging buffers you use (explicitly or not, drivers and runtimes are not generally shy of allocating behind your back).
The costs are basically latency-time-based (time to prepare the system for execution) coupled with execution-time-based, rignt now. The thing is they're done by measurement, not prediction, so even if I manage to screw up the setup and do something non-optimally, that measurement is still a valid discriminant for the system it's actually running on - and recall that everything is done at runtime, not compile-time, so it's that system that's being characterised.
I do actually update the network latency timings on an ongoing basis - there's a heartbeat that sends back info on a periodic basis, and the error-condition mechanism has to cope with a node falling off the network etc.
And again, thanks for the insight :)