That is: a lot of distributed big data processing tasks don't need to be distributed. Perhaps with beefy enough matrix multiplication chips, a lot of "big ML" tasks won't need to be distributed either.
That is: a lot of distributed big data processing tasks don't need to be distributed. Perhaps with beefy enough matrix multiplication chips, a lot of "big ML" tasks won't need to be distributed either.
At the end of the day I do believe ergonomics are going to win out. I think that's in large part why pytorch won out over tensorflow and jax; it provided the just-in-time computation that would allow people to more easily find bugs & visualize results without having to `compile()` everything down to a static computation graph. Hardware seems like a natural place for that abstraction layer - but maybe the silver bullet will really be on the software side, since we already have too many "low-RAM" equivalent ML devices in the wild. Cheaper to string things together after the fact vs. shipping net new hardware.
For instance, right now there's a new crop of "linear RNNs" (RWKV, Mamba, retnet, etc.) claiming to be as good as Transformers for language modeling but with two advantages: their compute cost is O(n) instead of O(n²), and they don't need to keep past context in memory.
I don't know if these linear RNNs will actually supplant Transformers, but I do think hardware requirements are likely to change over time.
We don't have overcomplicated distributed training infra. They are pretty much as complicated as needed.
Scaling vertically (having a bigger chip) is very hard. There are tons of tradeoff when making the chip, and it's overall an insanely complex problem. That's why Cerebras is a 8 years old company and yet you would be hard pressed to find anyone using them still.
And even if you give me a Cerebras chip that works perfectly, it will still be much easier for me to buy two of those chips and link them together in distributed training mode, than it will be for Cerebras to build a chip that is 2x the size.
The scale of the current generation of clusters to train models the size of GPT-4 are in the range of 25,000+ GPUs with 80GB of memory each, so no matter your chip size, complicated distributed infra is a necessity. Even assuming everything on Cerebras' marketing page is fully accurate, you would still need to distribute the training over 500+ of those massive chips to replicate a 25k GPU cluster.
However, the very next question a researcher will ask once a model fits on one device is “can I make it twice as fast/big if I use two?”
Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".
If everybody's job was just to do dot products all day I'd hope the answer would be yes.
I think the broader point is that the last x0 years of ML research show that more compute is better, both for iteration speed and for resulting performance. Distribution is just the natural outgrowth of that imperative once it reaches the limit of a single device/node. If Cerebras can address models at today's scale on one device, the immediate next step is "what can N of these devices do together to build models at tomorrow's scale"
I still think work on improving single-core/device performance is worthwhile, as distribution will always strictly not-better, and almost always strictly worse, due to coordination costs reducing efficiency. If two Cerberas can be glued together and achieve roughly 2x of their performance, it's still going to be more efficient than achieving equivalent performance from many more regular GPUs. Getting the hardware fast enough so that you need just one device for your problem - that's a special case that will yield extra win.