Rumours are that the original base model was a 16-way mixture of experts over 1 TB in size. With quantisation that would fit into 8x 80GB cards = 640 GB total. That is similar in size to competitive models. There’s also a practical benefit of being able to use 100% of a common platform such as a DGX server. Splitting across multiple servers is much more complex and less efficient.