Hacker News Reader

Comment by gosen | Hacker News Reader

> Per David Abitbol, who leads AI ops there: 56 Nvidia B200s, a blended 30 cents per million tokens, roughly a tenth of what they paid Anthropic and OpenAI.

So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling). I wonder how much did they spend on tokens to decide to make such a switch. I mean if they used to spend $500k-1M per month just on tokens then it would pay off in a year or so, assuming their infra will be stable and efficient.

We'll see if that's a trend for mid-big companies or just an experiment.

All that math is assuming that it's all self-hosted in their data center or so and not paying to a cloud provider, which will be probably more expensive.

View on HN

I doubt local will be all that common aside from regulatory requirements but buying them off your cloud provider will be the new norm for many enterprises.

The slice of the pie where frontier models are required is just gonna be smaller and smaller. And at point where AI wrote docs, tests, and all the easy and medium difficulty functions you are already saving a ton per employee

> So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling).

Are you assuming that they bought the hardware? because they could also be renting the metal off a provider, in which case they either pay by the hour or have a yearly commitment to save on costs.

It says in the video that they run them on Together.ai

yeah, seems like they're renting (according to the info from the video), but doesn't it make the bill worse? I mean the provider makes money somehow, so it should be more expensive for revolut. I guess it's a lot anyway

Maintenance, capacity management and spares all add up, not to mention labour. If they have a requirement to maintain geographical diversity then that moves the needle significantly if they’re primarily based on the other side of the country.

I've quickly checked and guestimate of the rental would be ~250-300k per month for this kind of usage they claim to have. It's all based on open prices from together.ai. So probably it worth the switch for them afterall, I can imagine that 56 engineers can burn tokens for more than 300k per month

It is in the article: "the gains showed up when tokens stopped being rationed" - so it is not just a "cost per token before/after" calculation. Once you told your employees "token are free, flatrate for all - go nuts!" they started to use it for all sorts of things and increased productivity. That is not easy to calculate.

Unless you have very strict requirements, it does not make any sense to invest in hardware. Most companies just rent it. Also, it's pretty hard to get the hardware.

The B200 was launched in 2024 if I'm not mistaken. The big gamble with this might be how fast these will deprecate in value/use as soon as Nvidia launches a new and better product.

At the moment at least Nvidia doesn't have a incentive to release a new product because it's sold out as it is and probably doesn't want to plunge their customers in depth with outdated products?

They do want to launch new product - Vera Rubin

In related news, Core 4 Solutions listed a decommissioned $60 million AI supercomputer cluster featuring 9,600 Nvidia A100 GPUs on eBay. It's six years old. Nvidia isn't Intel with tick and tock, but they're still going to come out with newer hardware every year.

The gamble in this case is pocket change. They could comfortably spend this amount every year and just use precious years for overnight / batch processing jobs when they become less useful for dev work but remain operational.

Personally, I think Anthropic and OpenAI will introduce enterprise plans such as $2,000/month/engineer plans that will give 100x more tokens than the current max plans and then charge API pricing.

Yes, margins will go down a bit but the frontier labs still have some leverage.

Reply on news.ycombinator.com