From what I can gather, DDR5 SDRAM are what data centers need (I assume it's involved with actual training, and efficiently running inferences from caches to consumers of their API's [?]).
VRAM, meanwhile, still remains the bottleneck for consumers. I think this means those working for Anthropic/OpenAI/etc won't be making as many meaningful advances to cut down VRAM for inferences. Why do thusly if that only allows the layman to use your open weight competitors in a more efficient manner?
For my critique on dataset and learning algorithm, that seems to only be pursued by European states who are lagging very far behind on LLM's. I'm happy that Nvidia is doing their best to be as open as they can with LLM's that the EU is acceptable to use, but there are still a lot of gaps in letting the layman verify end-to-end the entirety of the dataset[0][1][2][3][4] and the algorithms used to train the model outside of a high-level view of the framework used.
it goes back to international copyright law. Even if it was obtained in a legal manner, it's not okay to release it to the public. But it's fine to provide to states given you can prove the data was obtained legally.
So I'm not too hopeful for a decent open source LLM for the public to manage and improve on in the near future. Even if the datasets are actually open (which they're not, even for nemotron[4] beyond post-training), the architecture of learning algorithms with said limited dataset will still be proprietary. Just enough is open to be performant and compliant for government use.
[0] https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
[1] https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
[2] https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
[3] https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
[4] https://docs.nvidia.com/nemotron/latest/nemotron/super35/REA...