Hacker News Reader

Sure, they have clusters. We often call them "closet clusters". Nothing has changed. Some institutions have larger systems (some extremely large) but none of them have demonstrated running warehouse-scale systems. I'm talking one to two orders of magnitude (and the storage and networking to make sure all those systems don't stall waiting for data).

View on HN

Perhaps I am underestimating the size of these frontier models then, how big of a cluster do you think would be required to serve a university?

I don't think you're taking this thread seriously enough to respond in detail. A university can be tiny, or it can have thousands of researchers. Some of them might want to work on small models, and others on huge frontier models. Multiple groups training their own models at the same time. Combined with the storage, networking, power, and redundancy, you basically need large data center scale. And at that point you're basically throwing a lot of capital trying to compete with the hyperscalers; you might be able to serve a small number of researchers very well, but most of the consumers would end up unhappy.

I'm aware cluster sizes vary, I was mostly wondering how long much compute you need to serve a frontier model (since you seem knowledgeable on this topic). For instance back of the enveloppe Kimi K3 fits in ~24 H100, so my naive first impression is thats its not out of reach for a university-sized cluster to serve a few instances. I agree it may not make much economic sense I'm asking out of curiosity.

Reply on news.ycombinator.com