brrrrrm

It's worth noting that these pre-built blocks are both convenient and incredibly flexible. Before I started getting my hands dirty, it was hard to really see this. For example, here's a masked gather operation along the innermost dimension of an arbitrary rank tensor A:

    A[..., idx[mask]]

I'd hate to have to express that in serial code and then pray a compiler could lift the correct vectorized version out of it.

I've come to the personal conclusion that the issue with compiled languages and machine learning is that the API surface is so big and the operations themselves are very self-contained.

I find that I am not typically writing complex "language-level" logic that would be expressible in any type of language and would benefit from the performance of compilation. I am guessing APIs and quickly testing ideas in totally unrelated colab notebooks. My productivity goes up when I can quickly answer questions like "how does this transpose impact the shape of the output" or "how do I find the mean of this tensor along only these dimensions."

Compiled languages, and especially Rust, really require you to go in with a plan. I find most of the machine learning code I write is inspired by rapidly playing with ideas.

Without meaningful discussion of energy saving (flops per watt compared to Nvidia), this article is just an ad.

Pretty standard practice at this point right?

I'd start with the example of implementing the fastest reduction you possibly can. Pretty much all complexity in every kernel used in ML extends from this concept (reductions with addition).

https://developer.download.nvidia.com/assets/cuda/files/redu...

driving multiple GPUs on the same node is better handled by threads. Python is forced to use multiprocessing

Developer velocity is extremely important at FAANG scale.

The conversions are just a mask and shift? Super cheap

Some cool examples, like an animated attention head layer: https://bhosmer.github.io/mm/index.html?0=out+%3D+%28attn+%3...

AMD supports the 256 wide subset of the AVX-512 ISA?

I doubt it was obvious scaling up would magically work. I suspect the experiments were limited for analytic simplicity rather than computational.

I think you're more likely to train a model capable of writing Doom using this machine.

layer of indirection introduces scatter/gather and other dynamic loads, which is tricky to optimize

I wonder if the parent comment was made tongue-in-cheek :)

Once fused into super-instructions/adaptive instructions, does `disas` still allow users to manipulate the bytecode as if it were not optimized?

That'd be a very nice feature and I'm not sure it exists in other languages.

That’s very unlikely, tokenization is really simple and usually quite fast (scales with input size). Unexpected slowness with a larger context window might point to a non-existent or unoptimized KV cache.

nothing really, that's effectively what's achieved by comptime.

it's funny to see it written as "basic" dependent typing, considering it's an extremely complex type system to implement.

I've been working on mix of ML performance and abstraction for a while and this is a breath of fresh air. Zig is truly a fascinating language, and the semantics revealed in this post are surprisingly powerful.

"By hard coding the forward and backward methods at comptime we have some more comfort with the general correctness and expected errors we would receive if we passed in incorrectly shaped data at runtime."

This solves the issues of shape checking incredibly cleanly. Python libraries have been struggling with this for a decade. It seems like you could also extending comptime usage to also calculate allocations ahead of time.

Honestly, this whole thing makes me want to invest quite a bit of time into using Zig. Great post!

In terms of production pipeline, sure. But since there’s an established model architecture, the hardware doesn’t need to be super generic. Google had to build TPUs in a world where everything was still rapidly changing - deciding on things like precision, memory bandwidth, SRAM size etc. Microsoft could theoretically stamp out some GPT ASICS and call it a day.

> WebGPU ML runtime

oh cool! will this be numpy-like or will it have autograd as well? We're looking around for a web backend for shumai[1] and the former is really all we need :)

[1]: https://github.com/facebookresearch/shumai

ML is still in a compute hungry growth phase. Every exciting new achievement in accuracy requires more compute than before. It makes sense that tinyML, which has less compute, would always have lagging modeling support and be a bit less interesting.

TinyML does result in much cheaper products, though.

The WebGPU demo mentioned in this post is insane. Blows any WASM approach out of the water. Unfortunately that performance is not supported anywhere but chrome canary (behind a flag)

> JavaScript isn't used for any machine learning tasks as far as I'm aware

https://github.com/facebookresearch/shumai

My claim is pretty subjective, but the idea is that there aren't many distinct kernels used in machine learning. It's all tensor contractions and element-wise operations. I'd argue that this can be maintained by hand without need for automation or high level abstraction.

Triton is used in a templated way for a very specific albeit pervasive hardware (PTX compatible GPUs), which is why it works so well. Here's some of the code: https://github.com/pytorch/pytorch/blob/a66625da3bcdf1e262dd...

Generalized kernel generation (i.e. synthesis of optimal performance from non-expert user defined kernels and novel hardware) would be fantastic to have, but it just doesn't seem particularly necessary in the field.

> Hand-written assembly kernels don’t scale!

I used to think this. And I think, in theory, it is true. But the fact of the matter is, modern ML just doesn't use that many kernels. Every framework uses the same libraries (BLAS) and every library uses the same basic idea (maximally saturate FMA-like units).

Large language models are being run natively on commodity hardware with code written from scratch within days of their release (e.g. llama.cpp).

From a conceptual standpoint, it's really easy to saturate hardware in this domain. It's been pretty easy since 2014 when convolutions were interpreted as matrix multiplications. Sure, the actual implementations can be tricky, but a single engineer (trained in it) can get that done for a specific hardware in a couple months.

Of course, the interesting problem is how to generalize kernel generation. I spent years working with folks trying to do just that. But, in retrospect, the actual value add from a system that does all this for you is quite low. It's a realization I've been struggling to accept :'(

> be tied to aspects of the company's ... Coinbase Earn

"Commodity linked securities are investment instruments or securities that are linked to one or more commodity prices. Unlike commodities, which provide no income to the owner, commodity linked securities usually give some payout to holders."[1]

[1] https://corporatefinanceinstitute.com/resources/commodities/...

I don't think a high level representation is necessary for relatively straightforward FMA extensions (either outer products in the case of Apple AMX or matrix products in the case of CUDA/Intel AMX). WebGPU + tensor core support and WASM + AMX support would be simpler to implement, likely more future proof and wouldn't require maintaining a massive layer of abstraction.

It’s pretty obvious that cloning ML APIs (which are a collection of well defined independent math operations) is a better long term solution than shoehorning an entire language into the browser.

Hmm, slower as the number scales up. Wonder why my M1 didn’t do that

JavaScript is 10-37x faster out of the box without any imports

    async function time_tasks(count=100) {
      async function nop_task() {
        return performance.now();
      }

      const start = performance.now()
      let tasks = Array(count).map(nop_task)
      await Promise.all(tasks)
      const elapsed = performance.now() - start
      return elapsed / 1e3
    }

    for (let count = 100000; count < 1000000 + 1; count += 100000) {
      const ct = await time_tasks(count)
      console.log(`${count}: ${1 / (ct / count)} tasks/sec`)
    }

Outputs (Python 3.11, Bun 0.5.1):

    % bun textual.ts
    100000: 3767797.000743159 tasks/sec
    200000: 9001406.4697609 tasks/sec
    300000: 8281002.001242148 tasks/sec
    400000: 10038491.340232708 tasks/sec
    500000: 8976653.913474608 tasks/sec
    600000: 10437550.828698047 tasks/sec
    700000: 9443895.154523576 tasks/sec
    800000: 11021991.118011119 tasks/sec
    900000: 9790550.215324111 tasks/sec
    1000000: 10263937.143648934 tasks/sec
    
    % python3 textual.py
    100,000 tasks   303,063 tasks per/s
    200,000 tasks   270,058 tasks per/s
    300,000 tasks   271,621 tasks per/s
    400,000 tasks   261,945 tasks per/s
    500,000 tasks   251,070 tasks per/s
    600,000 tasks   272,520 tasks per/s
    700,000 tasks   250,977 tasks per/s
    800,000 tasks   253,131 tasks per/s
    900,000 tasks   244,696 tasks per/s
    1,000,000 tasks   266,061 tasks per/s