Is there a reason GPU's don't use insane "blocks" of sdcard slots (for massively parallel io) so the model weights don't need to pass through a limited PCI bus?
You can buy vertical microSD connectors, so you can stack quite a lot of them on a PCIe card. Then a beefy FPGA to present it as a NVMe device to the host.
Goal total capacity, as you can put 1TB cards in there. And for teh lulz of course.
Physically easy to build, main challenge for me would be FPGA implementation.
For a thought experiment that's not even optimized for speed, that's pretty good.
These are made similarly to HBM but are lower power and much higher capacity. They can also be used for caching to reduce costs when processing long chat sessions.