Transformers are typically memory-bandwidth bound during decoding. This chip is going to have a much worse memory b/w than the nvidia chips.
My guess is that these chips could be compute-bound though given how little compute capacity they have.
My guess is that these chips could be compute-bound though given how little compute capacity they have.
I agree that compute is likely to become the bottleneck for these new Apple chips, given they only have like ~0.1% the number of flops
FWIW I went the route of "bunch of GPUs in a desktop case" because I felt having the compute oomph was worth it.
Not in case of language models, which are typically bound by memory size rather than bandwidth.