Hacker News Reader

The title says "on commodity GPUs" but the only GPU mentioned (and the only one with benchmarks) are Nvidia H100s ($30K+ on Ebay)?

Do these run on actual commodity GPUs like RTX 3090s and what kind of tokens/sec is expected on those?

Also, there's no paper, no open weights, no code. Just an API?

Companies like Groq and Cerebras already hit these kind of numbers over a year ago so I'm not seeing what's HN worthy here.

View on HN

That's a good point. In this context, we've been using "commodity GPUs" to refer to standard Nvidia hardware, in contrast to specialized chips like Groq and Cerebras. While these chips also achieve fast speeds, they are not nearly as ubiquitous as Nvidia GPUs. We think that matching their performance on standard Nvidia hardware can make AI much more affordable. We also support any GPUs, not just H100's.

We're going to be releasing a tech report soon, stay tuned!

“Commodity” and “consumer” are not the same thing; H100 is commodity but not consumer, RTX 3090 is consumer and commodity.

An h100 is absolutely not a commodity product. A commodity product is one that is fungible (and that interchangeability is also reflected by second order effects like price). A 4090 is replacable with a 7900xtx. Not the same with an h100 and an instinct mi360

Reply on news.ycombinator.com