MLPerf training tests put Nvidia ahead, Intel close, and Google well behind

You seem to be suggesting that Google isn't focusing on performance on MLPerf but on something else that's more lucrative. Do you want to expound on what that is?

So... Its November 2023, and we have a handful of really good open source LLM base models (Mistral, Llama, MPT, Falcon, a few code models, a few Chinese ones like Yi, a Arabaic one) and a few notable proprietary ones, and a few cool vision models.

... Where is all that hoovered up accelerator compute going? Shouldnt we be swimming in open and closed LLMs by now?

It’s a bit like a shared space race. Getting that top-N accuracy to 74 point yada yada percent in 37 seconds on a certain resnet architecture tells you that you can do the same thing on that hardware, which means you can do your own things just as quickly. So it’s a worthwhile time investment. If you force everyone to get that same accuracy with the same model arch, you can make informed decisions, which is especially important when throwing millions of VC dollars around.

EDIT: Sorry, I completely misread you.

It’s difficult to quantify what you’d like to measure: the bottom line price of who is cheaper for your expected workload. There are immense trade offs, not least of which is time spent learning a particular (esoteric) stack. CUDA knowledge doesn’t port to JAX and vice-versa.

Prices are always changing, and you can usually work out a deal with the sales team to get lower than advertised. Especially if they want you to choose them instead of some competitor. So in general it’s hard to figure out what you can expect for production workloads in terms of total dev cost vs price vs speed.

I will say that as a researcher, there’s no substitute for fast iteration cycles. I’m one of the few who believe in scaling down your models as much as possible when testing experimental ideas, precisely because you can try 30 runs instead of 3. So all else being equal, I’d take speed.

But all else isn’t equal. The only thing I want nowadays is free plus stable. It’s looking like a 4090 might be the way to get that, which is enough to try out some interesting ideas.

Salaries more often than not, costs a hell of a lot more than compute and energy bills cost. If you have a 10k cluster, you're probably a mega-corp running a few teams sharing the resources running experiments in parallel to each other and themselves.

Now this is an very much so an overestimate on the actual cost of compute, the $10/hr/GPU cost is in all likelihood still significantly cheaper than the cash you pay to your researchers, and all the things that are needed to support them. Unless your team of researchers is about 10 people running very efficiently and making use of ALL the GPUs each hour of every day.

And just because the article doesn't mention the aspect you care about does not make it a clickbait.

[1]: https://the-decoder.com/nvidias-h100-gpu-sells-like-hot-cake...

https://blog.google/technology/ai/9-ways-we-use-ai-in-our-pr...