How would ML benefit from async? I thought everything interesting in things like TF or PyTorch is done by native code which is free to use multithreading and anything at all.
(I know little about ML.)
(I know little about ML.)
Right now the typical paradigm is this really clunky lock-step across nodes waiting for each other (everything is blocking). If your hardware is non-homogenous (both interconnects and accelerators) or your program needs to be run in a pipelined way, you're fighting an extremely uphill battle.
Go, JavaScript etc are the usual languages of choice for this type of stuff because it's easy to express non-blocking code.
That would be closest to true multithreading.
const y = await remote_model_a(x) // different machine
const z = await remote_model_b(y) // different machine
await z.backward()