Hacker News Reader

brrrrrmon Statistical vs. Deep Learning forecasting methods

I'm heavily involved in this area of research (getting deep learning competitive with computationally efficient statistical methods), and I'd like to note a couple things I've found:

1. Deep learning doesn't require thorough understanding of priors or statistical techniques. This opens the door to more programmers in the same way high level languages empower far more people than pure assembly. The tradeoffs are analogous - high human efficiency, loss of compute efficiency.

2. Near-CPU deep learning accelerators are making certain classes of models far easier to run efficiently. For example, an M1 chip can run matrix multiplies (DL primitive composed of floating point operations) 1000x faster than individual instructions (2TFlops vs 2GHz). This really changes the game, since we're now able to compare 1000 floating point multiplications with a single if statement.

brrrrrmon WasmEdge

what's the deal with wasi-nn? (who is supporting it?) Seems super high level and somewhat brittle

brrrrrmon JupyterLite: a JupyterLab distribution that runs in the browser

Why do you say that? I assume the timeit magic reduces a lot of overhead

brrrrrmon JupyterLite: a JupyterLab distribution that runs in the browser

it's neat, but it's kinda slow (25x slower than native when I tested it)

In a JupyterLyte notebook:

    import numpy as np
    a = np.random.randn(128,128)
    b = np.random.randn(128,128)
    %timeit a @ b
    1.5 ms ± 3.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

On my command line:

    $ python3 -m timeit --setup 'import numpy as np; a = np.random.randn(128,128); b = np.random.randn(128,128)' 'a @ b'
    5000 loops, best of 5: 59.4 usec per loop

brrrrrmon Machine learning with Unix pipes

Thanks for flagging!

brrrrrmon Python Asyncio

it's not a matter of performance but expressibility.

  const y = await remote_model_a(x) // different machine
  const z = await remote_model_b(y) // different machine
  await z.backward()

is trivially pipelined when run in parallel. With multithreading suddenly the backing C++ library has to be aware of this and figure things out for you

brrrrrmon Python Asyncio

Specifically distributed would be helped (multiple nodes).

Right now the typical paradigm is this really clunky lock-step across nodes waiting for each other (everything is blocking). If your hardware is non-homogenous (both interconnects and accelerators) or your program needs to be run in a pipelined way, you're fighting an extremely uphill battle.

Go, JavaScript etc are the usual languages of choice for this type of stuff because it's easy to express non-blocking code.

brrrrrmon Python Asyncio

All of distributed machine learning would benefit from async, but it's unfortunately a Python-only ecosystem for the time being. Probably gonna be a couple years before that has a chance of changing

brrrrrmon Python Asyncio

what do you mean "block"? If you call an async function from a non-async context it doesn't block.

brrrrrmon Python Asyncio

I think the issue is more like this:

1. You jump into a huge codebase and find a very useful function you want to use in your code (it uses asyncio)

2. Your code doesn't use asyncio, so you start converting functions to be async (or else you can't use the `await` keyword)

3. It turns out your function is called by a bunch of different users, some of which do not use the asyncio runner

4. You end up having several meetings with upstream teams asking them to use the asyncio runner

5. A single team says "no"

6. Scrap the project and just rewrite that original function you found synchronously

brrrrrmon Python Asyncio

I feel the same way, it's a lot like `const` in C++. Basically it's enforcing usage for every upstream user to ensure there's no inefficiency (such as waiting for things like the network).

JavaScript has an extremely nice "out" (I'm sure other languages do too):

    async function() {
      await async_thing()
      do_sync_stuff()
    }

can be written

    function() {
      async_thing().then(do_sync_stuff)
    }

which is quite intuitive and helps glue code together. I think the callback-based thinking really clarifies user intent ("just tell me when it's done") without impacting performance.

brrrrrmon Python Asyncio

Important to note:

> They are suited to non-blocking I/O with subprocesses and sockets, however, blocking I/O and CPU-bound tasks can be used in a simulated non-blocking manner using threads and processes under the covers.

If you're using it for anything besides slow async I/O, you're going to have to do some heavy lifting.

I've also found the actual asyncio implementation in CPython to be slow. Measuring purely event-loop overhead (and doing little/nothing in the async spawned tasks), it's 120x slower than JavaScript on my machine. https://twitter.com/bwasti/status/1572339846122991617

brrrrrmon Loop Optimizations: taking matters into your hands (2021)

In my opinion performance optimization should be taught from the ground up, rather than with lists of "performance rules." I found learning this stuff that a solid mental model of the compute resources available (registers, cache) and the paranoia of a compiler (correctness) often made it easier to write fast code. With a good mental model, rather than memorizing rules, one can just walk through the code and re-derive them from first principles.

I also find visualization to be extremely important in this domain: https://jott.live/html/mm_impl_anim

brrrrrmon Tinygrad: A simple and powerful neural network framework

Yep! This is an assumed optimization when it comes to modern linear algebra compilers. New primitives go way beyond FMAs: full matrix multiplies on nvidia/Intel and outer product accumulates on Apple silicon. It’s also expected that these are used nearly optimally (or you’ve got a bug).

brrrrrmon Tinygrad: A simple and powerful neural network framework

avoiding writes to memory and reducing the number of loops (although not FLOPs)

    for j in range(10):
      c[j] = a[j] + b[j]
    for j in range(10):
      d[j] = c[j] * 2

becomes

    for j in range(10):
      d[j] = (a[j] + b[j]) * 2

brrrrrmon Tinygrad: A simple and powerful neural network framework

> tread off the beaten path things get slow

Yea, makes sense. I think there's something to be said for dynamic compilation solving this problem more elegantly than providing tons of hand-tuned kernels (PyTorch is 890MB lmao https://pypi.org/project/torch/#files), but I don't think it's a strict reason for a performance win.

> change the loop order too

Memory layout as well! I'm 100% for dynamic compilation, but I'm claiming that it really finds its stride when you fuse things.

brrrrrmon Tinygrad: A simple and powerful neural network framework

The only thing I'd recommend is exposing "eval()" or something to let users tell you when they want you to evaluate things. It'll save a ton of time when it comes to hot-fixing performance and memory use issues. It's really hard to determine when to evaluate, and although it's a fun problem to figure out, it's nice to have an escape hatch for users to just tell you. (Flashlight has explored this and written about it here: https://fl.readthedocs.io/en/latest/debugging.html?highlight...)

If you're interested, I've looked into symbolic laziness, which allows you to infer correct input sizes even when the constraints happen later. Can be useful for errors. https://dev-discuss.pytorch.org/t/loop-tools-lazy-frontend-e...

brrrrrmon Tinygrad: A simple and powerful neural network framework

Doesn’t really matter for large batch/large model training on GPUs that don’t need much coordination.

But Python speed is one of the main motivations for a JS/TS based ML lib I’m working on: https://github.com/facebookresearch/shumai

brrrrrmon Tinygrad: A simple and powerful neural network framework

> It compiles a custom kernel for every operation, allowing extreme shape specialization.

This doesn't matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they're dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won't help much.

> All tensors are lazy, so it can aggressively fuse operations.

This matters. PyTorch teams are trying to implement that now (they have LazyTensor, AITemplate, TorchDynamo), but I'm not sure of the status (it's been tried repeatedly).

> The backend is 10x+ simpler, meaning optimizing one kernel makes everything fast.

The first part of that sentence matters, the second part doesn't. Kernels are already fast and their reuse outside of being fused into each other (which you need a full linear algebra compiler to do) isn't very high. If you make sum fast, you have not made matrix multiplication fast even though MM has a sum in it. It just isn't that easy to compose operations and still hit 80+% of hardware efficiency.

But it is easier to iterate fast and build a seamless lazy compiler if your backend is simple. You can pattern match more easily and ensure you handle edge cases without insanely complicated things like alias analysis (which PyTorch has to do).

brrrrrmon Tinygrad: A simple and powerful neural network framework

I've done some work in the past on NN representations and you actually can represent Conv and MatMul in more primitive ways. I ended up writing an IR called loop_tool that exposes this stuff:

https://github.com/facebookresearch/loop_tool/blob/main/pyth...

The idea is basically this: https://news.ycombinator.com/item?id=28883086

brrrrrmon The pool of talented C++ developers is running dry

You've presented a false dichotomy between "web" and "embedded systems".

C++ fuels the engines behind Python and JavaScript. Databases, compilers, machine learning libraries: all C++ and actively maintained.

brrrrrmon Python 3.11 is faster than 3.8

I feel like JAX usually comes up on PyTorch posts (in the same way PyTorch usually comes up in the comments on TF posts).

I think the delta between JAX and PT on commodity hardware is just a little bit too small to really create much splash on HN.

brrrrrmon Python 3.11 is faster than 3.8

most of the machine learning industry uses it almost exclusively as the driving language for GPUs

brrrrrmon Python 3.11 is faster than 3.8

I'll concede that Python has more nuanced bits of complexity exposed, but I suspect a willingness to break compatibility for the sake of performance would benefit the language a lot more than it would hurt it. I personally hope the hacker mentality and the immensely perf-driven nature of JavaScript is one day adopted by Python.

For those following, I want to make it clear that the "bits" being discussed are extremely nuanced and won't get in the way of doing obviously useful things in JavaScript.

Here's how you create a JS Object from C using Node-API: https://nodejs.org/api/n-api.html#napi_create_object

It conforms to ECMAScript 6.1.7 Object Types and is very much the same internal representation used by the VM.

Here's how one might tickle the internal representation used by Node.js (V8 in this case) to change the value of the underlying property of a JavaScript object from C: https://nodejs.org/api/n-api.html#napi_set_element

brrrrrmon Python 3.11 is faster than 3.8

You can expose objects. Here's how it is done in Bun: https://github.com/facebookresearch/shumai/blob/main/shumai/...

We've been using this feature heavily in Shumai.

I think you are vastly overestimating the complexity associated with this (user exposed ref-counting/garbage collection) and may not be totally up to date on what's implemented.

brrrrrmon Python 3.11 is faster than 3.8

how're you using it? what you printed out (modified.py) seems to be a different result than expected

brrrrrmon Python 3.11 is faster than 3.8

I don't buy this argument. WebAssembly is a vastly different beast than trivial C extensions (since WASM is a browser thing). JavaScript has both and is still much faster.

Bun (the runtime I used for the post), has a native C interface that's far simpler to use than Python (even with pybind/nanobind): https://github.com/oven-sh/bun#bunffi-foreign-functions-inte...

Deno (a v8-based runtime) has the same thing: https://deno.land/manual@v1.25.4/runtime/ffi_api

These use the standard C-ABI directly without requiring headers (unlike Python).

Node.js standardized a more Python-like C API called Node-API: https://nodejs.org/api/n-api.html

I believe Python can continue to add performance and is not hamstrung by the implementation.

brrrrrmon Python 3.11 is faster than 3.8

It's due to the runtime.

JavaScript has a runtime that updates as it executes. It notices a loop is being run frequently and then compiles a new version of the loop that runs faster.

Python doesn't have this functionality.

WebAssembly is for running things in the browser and would slow everything down (Python, JS, C++). The tests in the post are run natively (on my command line).

brrrrrmon Python 3.11 is faster than 3.8

I don't think there's much holding JS back from also being a glue language. JavaScript has certainly evolved from being only a browser language these days.

The only request I'd have is more granular garbage collection (marking FFI memory as refcounted for more immediate destruction).

brrrrrmon Python 3.11 is faster than 3.8

let me know if you find anything! Happy to edit the post with a correction.

For a little more information, this is using Bun 0.2.1, which uses JavaScriptCore (JSC), a part of WebKit which powers Safari. Since I'm running on an M1 Pro (Apple's ARM chip), there is probably somewhat of a benefit in using JSC.