Hacker News Reader

Moore's law, sure.

But rn it's totally doable to run a 65B or 100B model on CPU with a reasonable workstation.

Does Moore keep us from wiring more memory onto a GPU? Or making a GPU with expandable memory (slots?)

I think the biggest issue with VRAM right now is that it's very expensive compared to regular system memory, perhaps with the exception of the DDR5 for the latest CPUs.

But there should also be absolutely no issue in making a commercial TPU like google has internally for inference with more but less expensive ram and sell it. There surely must be a market now with these new models.

I think part of the issue is that consumers have never needed more than 12 GB of VRAM before. Game developers just don't have requirements that high. Crypto mining also doesn't require that much.

Now that there's clear demand in the hobbiest market for GPUs >100GB of vram, its more likely that manufacturers will step up with cheaper solutions.

Reply on news.ycombinator.com