Yes, it has a decent memory bandwidth (~750 GB/s) and it runs CUDA. But it only has 16 GB and doesn't support tensor cores or low precision floats. It's in a weird place.
Yes, it has a decent memory bandwidth (~750 GB/s) and it runs CUDA. But it only has 16 GB and doesn't support tensor cores or low precision floats. It's in a weird place.
Still, your point stands. It's crazy how that 2016 GPU has two thirds the FP32 power of this new 2024 unobtanium card and infinitely more FP64.
Is there a similar "magic value card" for low memory (2GB?) 8-bit LLMs?
Since memory is the expensive bit, surely there are low cost low memory models?
Standalone cards are more like dev kits.
(I’ve been tracking Tenstorrent for 3+ years and currently have Grayskull in ML test rig together with 3090)
The P40 was aimed at the image and video cloud processing market I think, and thus the GDDR ram instead of HBM, so it got more VRAM but at much less bandwidth.