All of distributed machine learning would benefit from async, but it's unfortunately a Python-only ecosystem for the time being. Probably gonna be a couple years before that has a chance of changing
(I know little about ML.)
Right now the typical paradigm is this really clunky lock-step across nodes waiting for each other (everything is blocking). If your hardware is non-homogenous (both interconnects and accelerators) or your program needs to be run in a pipelined way, you're fighting an extremely uphill battle.
Go, JavaScript etc are the usual languages of choice for this type of stuff because it's easy to express non-blocking code.
That would be closest to true multithreading.
const y = await remote_model_a(x) // different machine
const z = await remote_model_b(y) // different machine
await z.backward()