The flip side of this is if the GPU can access the main system memory then I could see this being useful for loading big models with much more efficient "offloading" of layers. Even though bandwidth between GPU->LPDDR5 is going to be slow, it's still faster than what traditional PCI-E would allow.
The caveat here is that I imagine these machines are $$$ and enterprise only. If something like this was brought to the consumer market though I think it would be very enticing.
(If anybody from AMD is reading this, I feel like an architecture like this would be awesome to have. I would love to run Llama 3.1 405b at home and today I see zero path towards doing that for any "reasonable" amount of money (<$10k?).)
Edit: It's at the bottom of the article. These are designed to be meshed together via NVLink into one big cluster.
Makes sense. I'm really curious how the system RAM would be used in LLM training scenarios, or if these boxes are going to be used for totally different tasks that I have little context into.