Democratize AI: turn a $95 chip into a 16GB VRAM GPU
medium.com
medium.com
The title is very clickbaity. What's the memory bandwidth of one of these compared to a gpu? Why 16gb of "vram"? Is that the limit of gpu accessible ram? What llm inference performance does it have?
Aim it at being able to run large LLMs from a single PCIe x16 slot at 75 watts.
A card that runs an LLM at 1/3 the speed is still worth almost as much to me as a an experimenter.
The processor(s) should be clocked down to not burn too much power or emit too much heat in my PC's case.
A version of the same card that connects over a very fast USB connection would also be nice. If it could be powered from a wall wart or USB-C PD, that would be awesome.
A P100 from 2016 (HBM2) has 732 GB/s memory bandwidth [1], whereas the ddr-4 [2] produces 50GB/s. So the AMD APU would be more than 10x slower at LLM inference.
Edit, even old GDDR5 cards like the P40 have bandwidth of 347GB/s. [3]
[1] https://www.techpowerup.com/gpu-specs/tesla-p100-pcie-16-gb....