Strongly agree. Bandwidth and latency are the two big limiting factors always though, and unfortunately they tend to improve slower than compute. I think right now the fastest infiniband you can achieve is 3-400 GB/s, and compare that to the H100 HBM3 memory bandwidth at up to 3 TB/s. So I don’t see a time in the near future where we can naively treat our server fleets as a truly unified machine.
On the other hand, work being done in the big three DL packages for making distributed training easier has been quite nice. I know the least about TF, but their dtensor looks promising. JAX’s entire distributed paradigm using pmap/xmap etc make certain classes of models very easy to distribute. The one I’m following most closely though is Pytorch and their sharded tensor, and it looks like they’re planning on implementing native distributed ops powered by their RPC framework, which should make full tensor- and model-parallelism significantly easier.