Hacker News Reader

Comment by arecurrence | Hacker News Reader

This is not as rare as public clouds may lead people to believe. I have had to move workloads around since AWS began (even between public clouds on occasion).

In particular, GPU availability has been a continuing problem. Unlike interchangeable x64 / arm64 instances with some adjustments based on the new core and ram count... if no GPU instances are available then I simply cannot run the job. AMD's improved support has increasingly provided an alternative in some situations but the problem persists.

I recommend doing the work to make the business somewhat cloud agnostic, or at the very least multi-region capable. I realize this is not an option for some services that have no equivalent on other clouds but you mentioned databases and k8s clusters which are both supported elsewhere.

View on HN

GPUs are better run in your own office.

All cloud providers charge much, much more for GPUs than if you run a local machine.

Cloud GPUs are also a lot slower than state of the art consumer GPUs.

Cloud GPUs: much slower, less available, much more expensive.

This is generally true for all accelerators (I work with cloud and on-prem FPGAs for my startup, Arbitrand).

However, lots of people only need those accelerators once in a while, so time sharing (aka cloud computing) makes a lot of sense and saves a ton of money overall. For FPGAs and some compute GPU applications, not having to handle support for your accelerators is also nice.

Say you want 100 GPUs all inter connected to your multi petabyte data lake that’s being fed by your production workload.

Sure you could buy all that equipment but I’d wager it’s cheaper, more agile, and greater velocity from it being in the cloud

I would argue that the cost profile is different.

Local GPUs are a big up-front cost. But assuming that your workload is stable, in the long run I think local GPUs ends up being cheaper per-hour than cloud.

For startups, it doesn't make sense to make the up-front purchase, fine. But if you're optimizing for long-term (amortized) costs, I'd be curious if cloud is cost-effective.

The long run being three months

An nvidia dgx box is roughly 40k. And that’s not including power/storage/rack Space.

But yes, If a single workstation can meet your gpu training needs then it’ll be cheaper with sufficient usage

For small orgs this makes sense, but this really depends on how big your data sets are that you're training against and how your ML Ops / Data Ops is set up.

GPUs are better run close to your data. If you're training on-prem then your data needs to be on-prem too.

You want to be in a position where you can spin up in a nearby region and pretend it's local and have things be good enough for a while. Properly building out multi-region is hard, and multi-cloud isn't worth it because it improves how you handle rare events (where half the internet is already down) with ongoing operational toil.

Reply on news.ycombinator.com