Anyone have comparisons to how many tokens/s a laptop with a mobile Nvidia gpu gets? Considering the cheapest M2 Max laptop in the US costs almost $3k, you can easily get a laptop with a 3090/4090 Nvidia mobile GPU.
However I suspect the unified memory architecture of the M2 is a big benefit here and this M2 Max system is either a 64 or 96GB model?
Even the very highest end Nvidia dGPU powered laptops won't have anywhere near that amount of GPU memory available and I don't know what the performance impact is with using system memory vs GPU memory on such systems?
M2 Max lands just around the 3060 mobile performance profile in this instance, it would be curious to see how the tokens/s reflect that.
Second, M2 Max might have a lot more RAM available than a 4090 because of its unified memory architecture. It can have up to 96GB vs 24GB for the 4090. So we need to look at different RAM configurations.
By Apple.
> unified memory architecture
Not really a bottleneck when layering over PCI exists. You're only constrained by PCI bandwidth and memory speeds, neither of which are really slow enough to meaningfully impact AI inferencing performance. Honestly, disk speeds are the #1 AI bottleneck I've seen on older systems.
> It can have up to 96GB vs 24GB for the 4090
Good, at the M2 Max's price point I could almost afford 4x 3090s anyways. I'd only need one to beat it in inferencing performance though.