LegNeato

VectorWare founder here. We are working with them and stoked they are investing more in Rust. I just gave a talk at RustConf about our different takes (https://rustconf2026.sched.com/event/2KNQj/making-gpus-feel-...). The video isn't up yet but you should check it out when it is. The efforts are complementary.

Just distracting / ugly to my eyes.

The goal of this work is to run existing unmodified CPU libraries (which may use core::simd) on the GPU. If you are manually writing ML-shaped workloads, it doesn't add any value over writing with tech like torch/tensorflow/jax which are custom built for those use-cases (except maybe familiarity if you are a CPU programmer).

It's just a way for us to add minutia and details that most don't care / need to know about. There are three audiences we try to make the posts accessible for: Rust people who don't know about GPUs, GPU people who don't know about Rust, and non-Rust non-GPU people. The toggle lets knowledgable readers go "wait, what about..." and hopefully the toggle answers it.

The goal is to use similar abstractions and code across both the CPU and GPU where it makes sense.

It is not currently available but we intend to make it available after we launch our products.

Part of our thesis is that decent GPUs are in every shipping device and most software doesn't use them and should.

The tentative plan is to open source all the compiler and `std` bits with our products built on top (compilers are not good businesses). More about our products coming in the next couple of months!

Didn't want to go into crazy detail in the post.

Each family of operations is a trait parameterized by the operation itself:

  pub trait EvaluateReduction<Operation, T>: LaneEvaluator {
      /// Reduce one distributed definition to an ordinary uniform scalar.
      fn evaluate_reduction(&self, value: LaneValue<Self, role::Distributed, T>) -> T;
  }

Call sites name the operation:

  let one   = evaluator.splat::<Splat, _>(1_u32);
  let two   = evaluator.splat::<Splat, _>(2_u32);
  let three = evaluator.binary::<Add, _>(one, two);

  let total   = evaluator.reduce::<Sum, u32>(three);   // a uniform u32
  let running = <Executor as EvaluateScan<Scan<Sum, Exclusive>, u32>>::scan(&evaluator, three);

Operations like Sum, Max, ReduceXor, Inclusive, and Exclusive are all distinct types.

As mentioned in the post, execution shape is typed too. A static shuffle takes its control as a type-level constant, and the shuffle mode constrains which controls are expressible:

  // Shift down one lane, keeping our own value where the source is inactive.
  let down  = <Executor as EvaluateShuffle<Shuffle<Down>, DownOrSelf<1>, u32>>::shuffle(&ev, v);
  // Broadcast from lane zero.
  let bcast = <Executor as EvaluateShuffle<Shuffle<Broadcast>, WarpLane<0>, u32>>::shuffle(&ev, down);
  // Butterfly exchange with the neighbor one bit away.
  let bfly  = <Executor as EvaluateShuffle<Shuffle<Xor>, Butterfly<1>, u32>>::shuffle(&ev, bcast);

For an example of errors caught, a warp-scoped executor for a device-scoped barrier is a compile error:

  <ScopedWarpExecutor<'_, WarpUniform> as EvaluateBarrier<Barrier<Device>>>::barrier(evaluator)
  // error[E0277]: the trait bound `Device: NvptxBarrierScope` is not satisfied
  //               help: the trait `NvptxBarrierScope` is implemented for `Warp`

Strip mining is typed on the amount of work and the lane capacity, and it hands back one chunk at a time along with the predicate saying which lanes live in that chunk:

  // Six work items across four active lanes: two chunks, based at 0 and 4.
  <Executor as EvaluateStripMine<StripMine, (WorkItems, ActiveLanes<StripMined<4>>), i32>>::
      for_each_strip_mined(
          &evaluator,
          (WorkItems::new(6)?, ActiveLanes::new(4)?),
          |index, active| {
           // ...
          },
      );

Hopefully that gives the flavor of it.

We never mention anything about superiority nor compare with other languages or programming models. This post is about making existing Rust CPU code work on the GPU.

Author here, AMA.

Oh, this again. I should put a website with this up...

I was the person who personally ran 10.6 security updates at Apple (10.6.1+), the "DRI". My team in the Updates Program office and I reviewed every single bug to determine if it should go in a security and stability update or wait for the next major version. Seriously, every morning we group triaged all Mac OS X bugs, both incoming and those nominated internally for us to look at and determine if it should go in an update. I packaged and audited the builds and tuned the delta vs full updates. I built the system that largely automated diffing "trains" for software updates (automastering).

The new version of the OS was always being developed in a branch/train, and fixes were backported to the current version as they were found. They weren't developed linearly / one after another. So, if you are comparing the most stable polished/fixed/stagnant last major version with the brand new 1.0 major version branch, the newer major is going to be buggier. That would be the case with every y.0 vs x.8. But if you are comparing major OS versions, Snow Leopard was different.

Snow Leopard's stated goal internally was reducing bugs and increasing quality. That is a fact, not marketing. I am not sure why people on the internet don't believe that, but I was there. If you wanted to ship a feature you had to get explicit approval from leadership and the bar was high. In normal feature releases it operated bottom up "here is what we are planning to ship" and in Snow Leopard it was top down "can we ship this?".

AFAIK Snow Leopard was the first release of this kind (the first release I worked on was Jaguar or Puma), and was a direct response to taking 8 software updates to stabilize 10.5 and the severity of the bugs found during that cycle and the resulting bad press. Leopard was a HUGE feature release and with it came tons of (bad) bugs.

The first .1 or .2 ALWAYS fixed critical bugs, because:

1. You had to GM / freeze the software to physically create the CDs/DVDs around a month before the release. Bugs found after this process required a repress (can't remember the phrase we used), which cost money and time and scrambled effort at the last minute and added risk. This means the bar was super high, and most "bad, but not can't use your computer bad" bugs were put in software updates...which was developed concurrently with the end of the main release (hence why .1 came out right away)

2. Testing was basically engineers, internal QA, some strategic partners like Adobe and MS, and the Apple Seed program (which was tiny). There was very little automated testing. Apple employees are not representative of the population and QA coverage is never very complete. And we sometimes held back features from seed releases when we were worried about leaks, so it wasn't even the complete OS that was being tested.

Software updates are always needed, though the issues they fix became less severe over time due to larger seeds (aka betas), recovery partitions, and better / more modern development practices. But I can tell you FOR A FACT that Snow Leopard had fewer major bugs over its lifetime, coalesced very quickly, and was extremely solid when Lion was released.

Quite honestly I would use cuda-oxide over rust-cuda for anything new right now....while rust-cuda is more complete, cuda-oxide is more aligned with nvidia's future direction (when we were rebooting rust-cuda we were told to by nvidia to stay with the nvvm layer but their plans changed and it is clearly not where their focus is). It's of course just rust so you could just feature-gate off differences as 80% will probably be the same if you wanted to try both projects.

cuda-tile is great...I would use it for anything written from scratch as it is closer to how the hardware (and projects like triton) work. But there are tradeoffs as it requries your code to be structured a certain way.

Maintainer of rust-gpu and rust-cuda here.

1. Rust CUDA is over 5 years years old, was dead for 2(?) years, but was rebooted and works. It enables both rust on the GPU and controlling the GPU from the host. The host library (cust) predates cudarc. If it was started today it would just use cudarc for the host side (and indeed, you can use the device side with cudarc). Rust-cuda is based on nvvm, which was the supported layer in the past but Nvidia is moving away from.

2. Rust-gpu (vulkan) works but isn't 100% complete of course. The dimforge folks are using, there are some crypto folks using, and we have llms written in it. Rust-gpu is only the "running rust on GPU part", the host is left to wgpu or ash or whatever.

The Nvidia projects are great (we gave pre-release feedback on them), but they were not announced when the call for presentations went out. They were also released as experimental with only 2 people working on them so Nvidia was cautious about marketing and over-committing.

We are the only people trying to bring rust natively to the GPU rather than just making the GPU work with rust (which others like cubecl, wgpu, cuda-oxide handle well). This has some interesting considerations from a rust language and compiler standpoint, which is what was talked about at the conf.

We sponsored an unconf room and got as many folks from all the various GPU projects together, including nvidia. The big problems right now are a) everybody has different needs, b) there are very few people working in the space c) the entire rust project is generally indifferent to GPUs. I expect this to change in the next year or two.

As a Puerto Rico resident, it is not "beyond time" and you have no clue what you are talking about.

See also the rust spir-v backend: https://github.com/Rust-GPU/rust-gpu/

There is no reason why compiler passes cannot be in an egraph, they are more general than optimizations. When you think about it, traditional compiler concerns like instructions selection are sort of a optimization problem if you squint.

Why do passes anymore when we have invented egraphs?

We are the maintainers of https://github.com/rust-gpu/rust-gpu and https://github.com/Rust-GPU/Rust-CUDA FWIW. We haven't upstreamed the VectorWare work yet as it is still being cleaned up and iterated on.

Agreed, and thank you.

It is not, we just haven't yet upstreamed everything.

It depends. At VecorWare are a bit of an extreme case in that we are inverting the relationship and making the GPU the main loop that calls out to the CPU sparingly. So in that model, yes. If your code is run in a more traditional model (CPU driving and using GPU as a coprocessor), probably not. Going across the bus dominates most workloads. That being said, the traditional wisdom is becoming less relevant as integrated memory is popping up everywhere and tech like GPUDirect exists with the right datacenter hardware.

These are the details we intend to insulate people from so they can just write code and have it run fast. There is a reason why abstractions were invented on the CPU and we think we are at that point for the GPU.

(for the datacenter folks I know hardware topology has a HUGE impact that software cannot overcome on its own in many situations)

Founder here.

1. Programming GPUs is a problem. The ratio of CPUs to CPU programmers and GPUs to GPU programmers is massively out of whack. Not because GPU programming is less valuable or lucrative, because GPUs are weird and the tools are weird.

2. We are more interested in leveraging existing libraries than running existing binaries wholesale (mostly within a warp). But, running GPU-unaware code leaves a lot of space for the compiler to move stuff around and optimize things.

3. The compiler changes are not our product, the GPU apps we are building with them are. So it is in our interest to make the apps very fast.

Anyway, skepticism is understandable and we are well aware code wins arguments.

No, it means not being able to see what is going on. Which is literally what the word blind means. You can be blinded by many things (blindfold, clouds/fog, bright lights, darkness, accidents, genetics, etc), permanently and temporarily. Non-humans can be blind and blinded. YOU are making it about a specific situation and projecting value judgements on it.

The author specifically says FLYING blind. Not "stumbling around like a blind person" or some such. If you are offended, that is on you. It's your right to be offended of course, but don't expect people to join in your delusion.

One of the rust-gpu maintainers here. Haven't officially heard from anyone at AMD but we've had chats with many others. Happy to talk with whomever! I would imagine AMD is focusing on ROCm over Vulkan for compute right now as their pure datacenter play, which makes sense.

We've started a company around Rust on the GPU btw (https://www.vectorware.com/), both CUDA and Vulkan (and ROCm eventually I guess?).

Note that most platform developers in the GPU space are C++ folks (lots of LLVM!) and there isn't as much demand from customers for Rust on the GPU vs something like Python or Typescript. So Rust naturally gets less attention and is lower on the list...for now.

Thank you! We're small so have to focus. If anyone from AMD wants to reach out, happy to chat.

Doing things at compile time / AOT is almost always better for perf. We believe async/await and futures enables more complex programs and doing things you couldn't easily do on the GPU before. Less about performance and more about capability (though we believe async/await perf will be better in some cases, time will tell).

Currently NVIDIA-only, we're cooking up some Vulkan stuff in rust-gpu though.

We aren't focused on performance yet (it is often workload and executor dependent, and as the post says we currently do some inefficient polling) but Rust futures compile down to state machines so they are a zero-cost abstraction.

The anticipated benefits are similar to the benefits of async/await on CPU: better ergonomics for the developer writing concurrent code, better utilization of shared/limited resources, fewer concurrency bugs.

Yes, that's the idea.

GPU-wide memory is not quite as scarce on datacenter cards or systems with unified memory. One could also have local executors with local futures that are `!Send` and place in a faster address space.