If the cloud was truly cost-effective then AWS and its competitors wouldn't be the cash cows that they are.
The cloud is cost-effective when your load is highly spiky, so you can spin up a ton of capacity during the spike, but not pay for it during the lull. (I worked in a company for which this was a principal case.)
If you are a relatively large company and want compute mostly for internal IT needs, especially for sensitive data, it may be cheaper to run your own server room and a private cloud. It won't be cheap in absolute terms though.
The fast scaling/up down is a good argument but you need to proceed with caution:
1) is your application actually structured to be able to do this? Would the overheads of developing your application in this way negate the savings (a fully distributed service with queues/etc has more development/management overhead than your basic monolithic MVC app, which may eat up all your savings)?
You need to do your own numbers here and be pragmatic. You'll have plenty of resume-driven people who see the potential of an engineering playground and will immediately tell you whatever it takes to make this happen, regardless of whether it will benefit the business.
2) how much does it actually cost to run your application at full scale outside of the cloud? I've seen places that do the whole autoscaling thing (and have lots more moving parts as a result) and yet a fully-scaled-up equivalent on bare-metal would cost less than the baseline scale on the cloud, thus removing the business case for it.
Keep in mind that the cloud effectively ate all the hardware improvements we've had in the last decade so switching to bare-metal hosting means you'll need a lot less hardware. A cloud "vCPU" is more like a pocket calculator when compared to a real CPU, and same with storage.
Is it? The marginal cost of setting up a old computer as a server is essentially zero.
I think the main benefit is that devs can spend $$$ on AWS without doing a procurement. I.e. AWS solves a beurocracy problem not a technical one.
You must know how to run a really scrappy startup!
All of a sudden, some post on Reddit brings in a wave of visitors and a large bunch of signups; your hardware barely manages. What's your next action? (Note: `kubectl scale` is not it.)
But it's more flexible and a lot of startups often pivot a view times.
Through if you have high compute and somewhat longer term predictable levels of compute then e.g. renting a rack in a server center and placing your hardware in can be notifiable cheaper (when your book keeping splits the initial buying costs across the years you run the hardware)
And then there is one huge lie about cloud, that it's way less work. It's not. It's a mess. I mean seriously we have a few docent years now of experience about what is user friendly and what is not when it comes to API/interface design and I would argue ~90% of cloud design from AWS is just plain terrible. That costs time, and time is money.
1) a lot of startups were never about building a sustainable business but to live the "startup founder" lifestyle and provide cushy jobs for your friends. In this case, complexity is a feature as it gives you an excuse to raise a lot of money, hire a lot of people and give those people tons of busywork to do (which they wouldn't have with a simpler stack).
2) even for startups who do want to build a sustainable business, VCs are often close-minded and anything that doesn't fit the typical "engineering playground" mold described above may fail technical due diligence despite it being better from an engineering point of view. So cloud and overengineering it is.
If your user base grows 100x in a year, there's no easy way to scale using your own hardware. But this is exactly the (rare) case the VCs are after. So a startup is required to build their tech in the cloud, even if in 99% of the cases it's an overkill and an extra expense.
Then there are also some highly managed/opinionated "cloud providers" (e.g. not aws), while they tend to have even more absurd prices for compute power they tend to actually safe quite a bit man power (in difference to e.g. AWS where the person time savings seam to be completely eaten up by subtle complexity and need for optimizing deployments in ways you otherwise would not need to). Which can make them attractive for very small startups which very low compute requirements. Through such startups also tend to not have calssical VC.
I have yet to see a startup for which that is the case, like at all. Idk if it's a SV thing but in most of the world this won't fly.
Most important you are missing the most important reason why startup go with the cloud.
Flexibility.
Startups often pivot multiple times in various aspects before they find a way approach substainability. Like common are changes in monetization model and changes in target audience. This often also comes with major changes in requirements for compute/hardware. This isn't just about amount of resources but also kind of them. Furthermore it also often comes with changes in architecture where services provides by AWS and similar can make such changes much faster/easier.
This means that for startups setting up their own hardware is a form of self-locking which startups can not always afford, and can even lead to a startup failing.
Naturally once a company has found a promising way to reach profitability the cloud vs. non cloud decision needs to be reconsidered.
IMO the simplicity promised does exist, it's up to the company how it gets implemented. Like either toolstack, if it's misused, it will cost more than they thought (and then they'll blame the toolstack)
Is there a well trodden way to serve most of your everyday traffic out of your own datacenter, but scale out to cloud infra when traffic peaks?
I think the analogy works best for SaaS but not necessarily for PaaS in most cases.
[1] if you're twice as expensive than a company that has say 10% margin, then with equal costs you should be with a 55% margin. And Amazon is more often than not more than twice as expensive as the alternatives.
but mainly, AWS and similar do not have a need to lower their raw compute price to that level because even without they get enough customers, e.g. there had been cases of Microsoft Azure in some locations running out of spare compute resources leading to issues for customer.
That's plain false.
most things compute related once you reach a certain minimal scale (i.e. need room cooling in your server room, etc.) get cheaper with scale sometimes quite massively
to list some
- hardware cost, sometimes you can't even get the hardware you need/want on smaller scale
- electricity cost (through it's complicated, if you can't choose the place for the data center electricity cost only up to some location specific scale)
- overhead for reliability (e.g. multiple power connections from different providers, multiple wired fast internet connections from different providers etc.)
- fast low latency internet access (what apartments get is often not cutting it at all even for quite small scale)
- man power for hardware maintenance (a single person can do hardware maintenance for quite a lot of units, having a team of such people makes handling sick cases, quitting and similar much easier)
- custom software tooling (amortizes more)
- physical access control (has a pretty high fix cost if you want to get it right, but after which it scales somewhat linearish, assuming we limit ourself to a single building or small complex)
- revenue also get's more regulare with more customers, which makes expensive unepxpected revenue drops due to e.g. a customer going insolvent much easyer to handle and as such cheaper
- hardware spare part handling gets easier and cheaper on scale
- ...
And sure there is an upper limit per-data center for scaling which is dependent on locality, but there are enough locations where this limit is quite high.
And sure if you run an such a small scale that you don't even have to consider having a "proper" server room with cooling system which is more then just fans moving heat out of racks you often also can cut many other costs, like power redundancy, network redundancy, 24/7 hardware fixing support, etc. Through then if you have some B2B services with reliability guarantees you often can't cut that costs no matter how small the scale (that btw. can be a reason enough to use the cloud, providing the necessary reliability with custom hardware on a small (compute) scale is very expensive as it involves many high fix costs)
Idk. if there was a misunderstanding so just to be clear, this is about cost of a data center per-compute unit getting cheaper with scale (up to some limit). Not about running your own ad-hoc reliability nightmare datacenter vs. renting cloud resources.