Hacker News Reader

Comment by abraxas | Hacker News Reader

We really need to one of the two things happen. Either the models are somehow able to run in regular CPU DRAM or we see the GPU makers to finally give us sensible amounts of VRAM. It's a travesty that a card I bought in 2016 still has more VRAM than many of the flagships being sold today. This card is 7 years old for goodness sake!

View on HN

I am really hoping we'll get something like AMD infinity fabric up and running with GPUs accessing system memory or something like that soon.

Then there is also moore's law :)

Moore's law, sure.

But rn it's totally doable to run a 65B or 100B model on CPU with a reasonable workstation.

Does Moore keep us from wiring more memory onto a GPU? Or making a GPU with expandable memory (slots?)

I think the biggest issue with VRAM right now is that it's very expensive compared to regular system memory, perhaps with the exception of the DDR5 for the latest CPUs.

But there should also be absolutely no issue in making a commercial TPU like google has internally for inference with more but less expensive ram and sell it. There surely must be a market now with these new models.

I think part of the issue is that consumers have never needed more than 12 GB of VRAM before. Game developers just don't have requirements that high. Crypto mining also doesn't require that much.

Now that there's clear demand in the hobbiest market for GPUs >100GB of vram, its more likely that manufacturers will step up with cheaper solutions.

Reply on news.ycombinator.com