brrrrrm

Puzzles + learning openings. I highly recommend lichess for this

what’s nice is that pyodide folks have gotten sympy compiled to web assembly, so it’s a viable browser library too!

Huge! Congrats team and Colin in particular :)

For large models, there are two main ways folks have been optimizing machine learning execution:

1. lowering precision of the operations (reducing compute "width" and increasing parallelization)

2. fusing operations into the same GPU code (reducing memory-bandwidth usage)

Neither of those optimizations would benefit from swapping to a faster language.

Why? The typical "large" neural network operation runs on the order of a dozen microseconds to milliseconds. Models are usually composed of hundred if not thousands of these. The overhead of using Python is around 0.5 microseconds per operation (best case on Intel, worst case on Apple ARM). So that's maybe a 5% net loss if things were running synchronously. But they're not! When you call GPU code, you actually do it asynchronously, so the language latency can be completely hidden.

So really, all you want in an ML language is the ability to 1. change the type of the underlying data on the fly (Python is really good at this) and 2. rewrite the operations being dispatched to on the fly (Python is also really good at this).

For smaller models (i.e. things that run in sub-microsecond world), Python is not the right choice for training or deploying.

> huge for the progress of WASM

HUGE! 4x speedup for anything parallelizable. This should bring around ~45GFlops out of the box in the iPhone browser, which isn't much but easily enough for basic neural networks (think object tracking, background segmentation, face feature tracking). For comparison, WebGPU would get that to ~1000GFlops

I believe they mean exploit the arbitrage opportunity to the point that the opportunity ceases to exist

My claim is that it’s not easy, not impossible. There’s little incentive to hack in JavaScript or maintain a Pyodide compatible build. The 4gb limit isn’t a technical limitation, just a standards thing (it could change easily).

the issue right now with Python support in WASM (at least for machine learning, the main driver of the language) is that Python is largely a wrapper language and none the utilities that make it so powerful (numpy, PyTorch, JAX) work particularly well in wasm, since it's so limited performance-wise (no FMA, no GPU support).

I'm excited for pairing wasm with WebGPU, which will likely unblock these projects from building support for the web/untrusted ecosystem. A useful project would be one that makes this integration really easy to build today and a flip of the switch to turn on in the future.

This is a really cool effort! I’m curious, where do the bottlenecks live right now?

There’s a C embed example in the examples folder

> If in the near future JS also reaches good support for scientific/ml libraries it seems like entering that ecosystem is an obvious advantage

Fully agree with this sentiment. To that end, I’ve been contributing to Shumai (linked by someone else in the comments) to enable torch-like machine learning for Bun

Storage isn’t much of an issue these days. The bottleneck is computing values from RAM, fp8 can improve this drastically

Mentioned in the post, Cornell provides some really cool interactive data:

- fine grained map of the wages/% making livable wage: https://blogs.cornell.edu/livingwage/

- likelihood of making livable wage based on a multitude of configurable factors: https://blogs.cornell.edu/livingwage/calculate-probability-o...

without using intrinsics? `-O3 -march=skylake-avx512 -mprefer-vector-width=512`

well, AMX does asynchronous outer products from byte-addressable buffers with accumulation. it's not like tensor cores or intel's AMX where the silicon literally only does a matrix multiplication. you can easily do efficient sliding window math from the buffers (convolutions) or sparse data loads (since the instructions run asynchronously from the CPU)

typically if it's available, compilers will use the avx512 register file. This means you'll see things like xmm25 and ymm25 (128 and 256 bit registers) and those are avx512 only. However, compilers using 512-wide instructions is kinda rare from what I've seen

I don't see throughput information on Intel's AMX, but their VNNI has information: https://www.intel.com/content/www/us/en/docs/intrinsics-guid...

0.5 cycles per instruction max, 3.7GHz clock, that's 7.4e9 instructions per second. If I'm reading it right, that instruction does 16 4-wide dot products, which is ~128 ops. So ~950Gops peak in int8 precision on a server class Xeon assuming no clock throttling.

(edit: flops -> ops)

I believe the vim.wasm is pre-compiled and checked in as a "binary." The javascript written is a runtime wrapper

author here! I grabbed them from here IIRC https://emojipedia.org/apple/ (there are a bunch of sites that have these)

edit: yep - predicting the original based on the augmented version (as a hack to generalize for unseen inputs)

> It's really a shame that data science, ML, and notebooks are so wrapped up in it. Otherwise we could jettison the whole thing into space

Although I personally feel Python has its place, I contribute to a project that hopes to diversify the ML/scientific computing space with a TypeScript tensor lib called Shumai: https://github.com/facebookresearch/shumai

I feel similarly. However, Python has

- reference counted objects with predictable scoped destruction. this keeps the maintenance of huge resources quite simple and memory usage steady

- operator overloading (infix, like +/*-)

I really wish other languages (like JS/Go) had this stuff.

I’ve found javascript is much better than wasm for lots of this stuff

https://bwasti.github.io/mebm/

if everyone on earth shared a memory space and every person had 4 gigabytes of byte addressable storage, we'd be at 65 bits.

does this support any form of FFI? It'd be nice if users could shim in lightweight APIs that clone libraries like numpy/pytorch -- it'd immediately make some machine learning super portable!

ChatGPT uses a modified version of GPT3 (details are sparse, but here are some: https://beta.openai.com/docs/model-index-for-researchers). There’s a logbook associated with the training of OPT (similar size to GPT-3): https://github.com/facebookresearch/metaseq/blob/main/projec...

This article estimated the cost of training GPT3 to be over $4M: https://heits.digital/articles/gpt3-overview

edit: TorchInductor calls into GCC/Triton depending. I mistook it for the various backends TorchDynamo supports (including TVM)

Some context/history:

For compiler people reading this, a lot of common compiler terms have been entirely reinvented in the context of machine learning frameworks. An ML "graph" refers almost exactly to the dataflow graph (DFG) of a program. TensorFlow 1.0 only exposed a DFG, which is well known to be far simpler to apply optimizations to (assuming you have a linear algebra compiler).

PyTorch integrated with Python (an interpreted language) and does not expose an underlying DFG. This is labeled "eager" and means that compilation of PyTorch requires optimization over both the control flow graph (CFG) and DFG. Python by default exposes neither of these things in a standard way. Some ML workloads simplify easily to a DFG (torch FX can handle this), but the general case does not. Although TorchScript (a subset of Python) tackled the CFG in 1.0, the team is now taking it further and compiling Python byte-code itself (with torchdynamo), which means you don't need to change any code and still get compilation speed ups! That's why 2.0 is significant.

Of course, all of this requires a linear algebra compiler to actually do the optimizations which is why things like AITemplate (for inference) and TorchInductor (which calls into a bunch of other compilers for training) exist for PyTorch. TensorFlow's linear algebra compiler is XLA.

I disagree, the burden is almost exclusively maintaining fast implementations of primitive operators for all hardware. These ML libraries are collections of pure functions with minimal interfaces. There's very little code interdependence and it's not particularly difficult to implement modern algorithms to train networks.

full stable diffusion in <800 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

autograd in <30 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

Adam in <20 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

A somewhat related, visual guide to discrete 2d convolutions: https://ezyang.github.io/convolution-visualizer/

that's a good point. I guess as an addendum it's not just compute efficiency but also "statistical efficiency" (if that has any meaning?)