The cloud backlash has begun: Why big data is pulling compute back on premises
techcrunch.com
techcrunch.com
As the current submission appears not to have a workaround, I've downweighted it.
When it's time to negotiate your enterprise discount program, savings plan(s), etc., etc. you need leverage, and that's what this is. Some companies spend multiple hundreds of millions of dollars with AWS alone and at that cost any and every tactic to bring down spend will get deployed.
We had a global network operator talk to us about repatriating its ML workloads on-premises from Azure. Cost is the main factor, sure.
I don’t like Azure and won’t use them again but this seems like a smart approach for MS to take.
https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap...
What percentage of COGS is your cloud/infstracture spend? If your products don't look like cloud infrastructure (aka Dropbox), it better be very small. If you think cloud is costing you 1000% what it would cost on-premise, that's almost certainly a spend and architectural discipline problem, not a cloud problem.
Whether that's an important saving or not changes from business to business, but anything less than 1000% seems only achievable by customized discounts from the cloud provider.
Still, I think it's hard to justify going anti-cloud in an effort to cut costs. Without a very skilled and experienced team doing the migration, the end result is very likely to be some hybrid setup that's the worst of both worlds. Especially if you're using a managed service, the open source alternatives might not be the drop in replacement that they claim to be.
Plus, most jobs today expect cloud experience. So there's an incentive for developers to push to use a cloud so they can grow into their next role.
This here is the compelling case for the cloud. It was always less about the features and the money save, it's that I now have 1 stop shop vendor, if I need a queue I can spin it up in 10 minutes, instead of filling out a ticket to internal IT getting them to put it on their backlog waiting until they have the cycles, and in the meantime I'm sitting on my thumbs.
I need a new server, in the old times it was a ticket, a budget meeting, two or three levels of sign off, a business case, on and on. Whereas in AWS or Azure I can spin up a box in about 20 minutes and be all ready to go, and happily start developing.
Just like MS was never about technical excellence, it was about being a pre-approved vendor that had everything you needed, so you wouldn't have to go through another vendor approval process. Same with the cloud now scaled down to a team size, I now know I am pre-authorized to spin up X number of resources without having to worry about security throwing a fit, or blowing an enormous amount of money.
Because companies do not like things going into assets category. Instead rent dedicated servers on Hetzner or wherever else it is price competitive and see red tape disappear.
>"So there's an incentive for developers to push to use a cloud so they can grow into their next role."
This is very lame argument. I am developer and a business and do understand both sides but sorry I run a business to make money, not to give them to Amazon and definitely not to be playground where developers can play with Amazon tech at owner's expense.
A company like uber is not looking at the price of a server but looking at the TCO of the products living in multiple verticals. TCO includes many things.
https://www.columbusglobal.com/hs-fs/hubfs/United%20States/B...
This is true, but it doesn't preclude use of the cloud.
The core of the cost issue is, in my view, that software is still designed to run in a datacenter: always on, all code in a single process (yes, this includes microservices), expected to never stop mid-execution. That made sense when you owned the hardware and needed to fully utilize it for the duration of its life. This prevents the best cost-saving measure in the cloud: turning it all off.
People can complain all they want about how the cloud providers operate, but there are ways to utilize cloud infrastructure without breaking the bank. Software development practices have to change, and in the meantime the cloud platform providers are going to make a pile of money as companies shift legacy software to the cloud without the needed re-architecture. We need frameworks and other technologies that, similar to the abstractions we now have for memory management, multi-threading, asynchronous I/O, etc. relieve the burden of reasoning around distributed, disjoint processes from the developer. Then we can finally realize the ideal of "only pay for what you need".
Can you give some example of that being the case?
However, the cost calculation is weird when you factor that 2 hours of cloud "on" can often pay for an entire day of "on" for a normal server.
All else being equal, which I'm aware is a contentious/controversial notion in of itself, even if I personally believe the colo costs for headcount are often greatly overestimated.
">People can complain all they want about how the cloud providers operate, but there are ways to utilize cloud infrastructure without breaking the bank"
Alternatively do what I do (example of particular product I developed and nor run for a client):
Single exe in C++ talking to a local Postgresql database. Combo is capable of processing thousands of requests per second sustainably without breaking sweat (this will cover business needs for a next 20 years). Also standby always on server with replication. Daily backup to on premises backup server. Hardware for production and standby is dedicated 16 Core AMD with 128GB RAM and 2x4TB SSD. Each rented from Hetzner at about $100 each. Fronted by Clouflare, not sure about the price as Client pays it. All deployment / management / maintenance / backup / Point In Time Recovery is done by couple of scripts. Reinstalling whole thing on a totally new server takes running single script for about few minutes (at some point will take longer as the size of the database grows but would still be very manageable).
The cost on Azure for the equivalent processing power and features accordingly to their calculator is 10 times more.
Most companies that save a lot by moving to the cloud are the ones which have several workloads, multiple products, teams, and verticals. They got security requirements, compliance, logging requirements to name a few.
So yeah, if you only consider a most basic business that has a single website with a single use case and non of the above then Hetzner is a great option.
You seemingly refer to a discounted mode of execution in AWS where your servers can be pulled off at any moment and there is no uptime guarantee. That only works because few use this mode. If everyone starts using it, the discount will disappear.
Whether it can be made cost-effective in practice or not, I don't know - I haven't made the calculations. Just saying that's how I read it.
We considered autoscaling web backends once. You've got to worry about capacity issues. You don't want a load spike only to discover EC2 has run out of the instance type you use. I've seen that issue 3-4 times in AWS now and it's painful when it causes downtime. Your app needs to have spikey load for the pricing to work, for us, even a strong day/night cycle didn't give enough "off time" to cover the higher hourly prices. Cloud vendors give steep discounts for always on usage. Your load also can't be too spikey, as spinning nodes up and down takes time. Making nodes start quickly can be a dedicated project.
I think we also looked into moving CI/CD servers to on demand, but some engineers worked too late into the night and others were up early. IIRC the "off period" was long enough to provide some cost savings, but not high enough to justify engineering cost.
Something like Lambda doing a "scale to zero" with a generous free tier looks cheap, but if you ever need to scale up you'll find it's quite expensive and your stuck with a weird architecture that's hard to move to something traditional. It is cool when an app fits the Lambda free tier, but that's not a good model for companies paying very large cloud bills.
The problem is that someone needs to do capacity planning, someone needs to buy servers. AWS has to pay the cost of an unutilized server by... keeping the hourly cost of on demand servers/cloud functions high.
Isnt it natural that your service gains more users at this time when everyone is trying to cut down on cloud costs... Cloud usage may be declining, but those who are still using it would want to cut costs...
Pre-COVID I was a at a F500 that had gotten screwed pretty hard by an Azure deployment than ended up costing TONS more than anticipated. Even after we made it clear it would be a lot of OpEx. "Sales engineers gonna play games" -- and they did.
New gig is just going down that path, and is, go figure, seeing the same problems. Burned by Azure, and now getting bilked for every last dollar by GCP.
Seconded. We called in Accenture to tell us if we should do the cloud.
They said Yes. I was on the Solutions & Architecture team, and we asked to take a look at their numbers.
Found multiple errors in their spreadsheets. Math being straight up incorrect, or calling out pricing that didn't match what our SaaS sales guys gave us.
"Oh they're old reports". Okay fine, run them again with the new numbers, and slap the recent graduate who you forced to make this excel.
Again, the numbers came back strongly in favor of Azure. Took a look again, this time had some fruity macro and pivot table, and we couldn't clearly make out what they did to get their final $$$ figures.
3 months later we declared we were "cloud first". There was much speculation as to how much $$$ the directors and VPs got. In a different life I was in sales engineering and there was an unspoken rule about "referral agent fees" to 3rd parties who help us close deals...
It's not really a 3rd party if they are a director at the customer...
With computers it's also usually the case that the Late Adopters show up in the middle of the Trough of Disillusionment of the Hype Cycle.
It takes a long time for new trends to percolate through the management class. I had a gruff old engineer at an early job who used to joke that one of the executives must have gone in for a dentist appointment and read the 5 year old tech magazines in the waiting room, because we have a new initiative to start doing something that other people are already stopping.
https://d2908q01vomqb2.cloudfront.net/77de68daecd823babbb58e...
In fact cloud is often complex, requires lots of highly specialsed systems management, is under powered and very expensive.
9 cents per gigabyte out if you’re lucky - that’s a startling price.
Machines are old and out of date and left behind.
Vendor lock-in is a real problem .. you probably can’t exit even if you want to.
And worse, clouds often don’t even have the computing resources you need with “requesting quota” and waiting for your “quota” to be granted some time in the next 24 hours, the opposite of elastic and scalable.
And if you need GPUs then the news for you is that gamers have vastly more powerful GPUs at vastly lower cost (even with Nvidia and AMD fleecing consumers on their next generation cards).
Rent a bare metal server from a second or third tier player. Or buy a couple of fast fiber connections and buy some 16/32 core machines with 128 gig Ram.
And honestly it’s not hard to run modern Linux machines, that’s an old crones tale.
Recently I had to do some performance testing. In the past I would have quickly started 5 or 6 instances of different types, run my performance tests, shut them down and be done within 2 hours.
Since AWS introduced zero quota GPU instances, I had to request quota for one specific GPU instance type in one region and then waited 24 hours till it was “approved”.
At that point I completely gave up on AWS as being a viable part of any GPU architecture.
It gets no less scalable or elastic than that.
I literally could have gone to the shop and bought a fleet of machines and built them at home in the time it took AWS to approve a quota of one GPU instance in one region.
That said, some hardware companies have APIs to do most of that stuff - a little human runs out to fulfill your request if need be in some cases I think. Still like... much to worry about.
Unless you don't want strangers touching your assigned rack, then you need to bring the key.
I knew the cloud marketing machine had done the new generation dirty. I didn't know it was this bad.
Edit: also OSHA rules, cable splicing, how to handle ladders, etc, etc.
If you buy colocation, it is tho.
So to your point it’s how the internet is run. But your midsize enterprise isn’t a telco. They have no expertise with this stuff.
For plain S3. Other methods have lower prices. All methods have automatic volume discounts and almost everyone has a free tier.
> Machines are old and out of date and left behind.
A problem I've seen on every single VM stack, regardless of where it is hosted.
> Vendor lock-in is a real problem .. you probably can’t exit even if you want to.
Build your model first, then map it into services, don't start with the services and map your model based on them.
> And worse, clouds often don’t even have the computing resources you need with “requesting quota” and waiting for your “quota” to be granted some time in the next 24 hours, the opposite of elastic and scalable.
I've only experienced this on services that have a potential for abuse. DNS and SMTP being all that I can remember.
> Or buy a couple of fast fiber connections and buy some 16/32 core machines with 128 gig Ram.
And "requesting quote" isn't a delay item here? I don't know anything about fiber, how to price it, or how to receive it... and the capitalization process on all this is going to eat up a bunch of my departments time.
> And honestly it’s not hard to run modern Linux machines, that’s an old crones tale.
It's much easier to just run a lambda. It's a great compute environment. It does 90% of everything I've ever needed.
I totally agree with that. I'm a dev, not a sysadmin. But I run several Linux servers and can set up the network(s), the firewall, the apps and all that. It's not that hard.
You can buy incredibly powerful machines today at a very cheap price. The 20 cores Asahi Linux ARM workstation, 128 GB of RAM recently posted here comes to mind... In the blog's author's word: "The Mac blew the AWS c6g.12xlarge Graviton instance with 48 ARM64 cores instance out of the water."
Or this slightly older blog entry about someone running 25 Gbps fiber to the home. Not to the office. Not to some colo. To the home. Sure, it's not available in many places, but that's quite something. That is someone, at home, having a personal pipe wider than what cloud instances do provide to most.
Heck, my brother, a poweruser (not a dev), just assembled a 7950X / 64 GB of RAM with a 4090 GPU. I think many, used to their laptops, don't realize how powerful such machines are (laptops aren't exactly known for running 4090).
It's honestly a bit insane to me that several devs have, at home, wider pipes and faster machines than what their apps are going to be deployed on.
I'm "only" at 2 Gbps FTTH and don't have a particularly fast machine (just ordered a 7700X / 32 GB of RAM / WD SN850X Black as suggested by someone else here yesterday) but even that is sufficient to make me question a huge lot of cloud instances offerings.
What were seeing is that some cloud native companies have reached a size and maturity where the on-premises move makes sense. That's all, no fundamental shift.
[0]: https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap...
I am participating in K8s migration to on-prem HW in customer's owned server room. The scaling flexibility is close to none. Old time HW procurement processes take months. Blades need to go thru HW and NW Engineer's hands for a few weeks, before even K8s workers can be installed there, then apps scaled. This is for a project which has known requirements and performance expectations. Forget any of that if you are just privoting start-up.
But for anyone who just wants to "own" the data, the stack and to squeeze the maximum performance, then bare metal dedicated hosting or rack rental should be enough.
But I also tend to favour hybrid setups these days, and most managed providers also has a cloud offering, which gets you the best of both if you have a decent provisioning setup and an overlay network:
The base load on managed servers at whatever commitment period gives you the best prices. The cloud instances provides elasticity. Done well your users/devs can't tell where one ends and the other starts. And because you have a cloud setup tied in, you can go pretty close to the wire on those managed servers - a lot more so than with a purely on-prem/purely colo/rented servers setup, increasing the savings from the managed servers even more.
I've had setups like this where we hardly ever needed to spin up any elastic capacity, but where having the capability meant the managed servers often got to 90%+ utilisation because we could let the utilisation rate get that high without issue knowing the system would spin up cloud instances if necessary.
For the parts of your business that are steady state, cloud can be dumb, if you care about spend. At any significant scale you can trade flexibility for control and reduce cost. The other benefit is capex vs opex and how that affects your taxes - cloud makes it easier to shift to a pure opex model, but it’s not an exclusive ability.
It depends on you however. It’s almost always a good deal to buy Office 365 or Google. If you have a heavy cyclical business, it’s probably smart to rent vs buy compute. Pay AWS October-December and turn everything off in January.
At the end of the day, any CTO who has religious beliefs about this stuff is an idiot. Execution and cash drive these decisions.
Plus there's some 8 fallacies going on here. The Cloud Providers generally make it very expensive to have some of your stuff in the Cloud and some on-prem. That's a challenge for most businesses, which I typically assert should be about 80% boring and 20% volatile/dynamic/interesting/on fire, because that 20% can soak up a lot of fees on chatter between the two classes.
That's by design. They want you all in or all out. They don't want you going back and forth. Whatever that was originally informed by (eg, the costs of their wires into and out of the Internet), that's where we are now and it seems to be considered a Good Idea.
If they change that they might keep some customers, but they'll also give others an easier exit strategy.
If managing cloud is skill-intensive and you need a hire a person/team with specific skills in it, then you're no different from on-prem from a personnel perspective.
And those are growing pains. In 5 years time, maybe the proper controls/cost containment will be easier because we've seen how easy it is to fuck it up.
We're comparing an ideal version of cloud to practical versions of on-prem. When the ideal version becomes practical, it might be a no-brainer. But for now, companies are going to have to do what's best for them today.
So you run your own services on their hardware (as opposed to using cloud SaaS, where you rent both the hardware AND the service).
Cloud is also so centralised that even if everybody collectively got better at optimising costs there's a serious risk that cloud vendors would just collectively fatten their margins. The inverse is also true, if on-prem suddenly became much easier to run, expect licensing and hardware costs to start creeping up from anybody with anything resembling a monopoly or cartel.
The problem is companies doing one thing have no idea how expensive is doing that other thing... grass is always greener on the other side.
I'm not really "out there" and "with it", but it seems the move has been away from classic OOP for a long time now. I never see new projects try to have the inheritance tree's that C# or Java have. I think small simple objects can be fine. It's when you get into metaprogramming with templates and multiple inheritance in C++ that I want to just run away.
I mean, what kind of CTO would you be if you didn't have a cloud project going on? Who would hire you if you got booted?
Someone who isn't an idiot that jumps on every bandwagon they see?
- lokalise
- newrelic/datadog
- opsgenie
- gitlab/github
- slack
- atlassian suite
- google/microsoft suite
- HR (e.g., personio)
- miro
- and a large etc.
These saas charge per month per user. As soon as your company has a couple hundred employees that require such platforms, we are not talking pennies anymore.
I think a lot of these SaaS companies are feeling the squeeze and pressure to be profitable. A few that we've been customers of since they were wee companies are jacking up rates by 3-5x. The value of an $8k product has a lot lower expectations than a $40k one.
Not to take a negative view, it's just an unsexy skillset and doesn't attract a lot of talent and attention.
Trying to hire for folks who can gracefully manage on-premise networks, power, and hardware is possible and extremely challenging. "Cloud," as it is today, is here to stay.
The primary focus of this purely infrastructure position, is a coding take home. Now, infra as code is part of local cloud, but there is sooooo much more to handle.
But because they have no idea how to handle infra, they seem stuck thinking it's just a dev job. Even while they hire stating it is not.
Quite sad. So much knowledge lost, that they don't even know what to look, what to hire for.
I've been hearing this "Cloud will DESTROY on-prem because on-prem skills don't exist and standardised cloud technologies make it easy to hire from anywhere across the world!" narrative for years which seems to deny the reality that we managed to train people to run datacenters just fine for decades even if people are snobbish about the riff-raff running things these days. Larger companies long ago figured out they could work with schools to develop a mutually beneficial training and employment pipeline. A lot of managers seem to be too lazy to do such things and find it easy to simply delude themselves into thinking cloud is always the best choice instead of contemplating doing hard work like that.
I used to be super-against cloud until I became the backstop for our entire IT infrastructure. It's really funny what happens to your beliefs when you take on ownership of more problems. Complaining about how Azure is 20ms slower than it needs to be on some storage service is nice, but are you prepared to personally take total ownership for that service's health by bringing it on-prem? Talking about how we could do it cheaper & faster in our house is super easy. Actually executing is a completely different story.
I won't even move back to on-prem Active Directory at this point. The convenience of having this managed centrally is unbelievable compared to where we were at a decade ago. Today, an employee can go to Best Buy, grab a completely new machine, connect it to our directory using the retail OOBE and start working productively within 20 minutes. Not one time does the employee need to bother an IT person during this process. Could you achieve the same with an on-prem setup? Absolutely. Would it take more money and time than we have available? Probably.
There are risks with going all-in on cloud, but I assert that those risks are almost always less severe than the risks endured with on-prem. When is the last time a cloud vendor made some last-minute change to a product or service that endangered their customer base? You typically have years to migrate off something a cloud vendor doesn't want to support anymore, simply because they have to make sure 10k other customers don't balk too. This is also why these products are so stable in terms of roadmap. Going with the herd is a proven way to protect yourself. Being the one shop still online when 99% of the clouds go down is probably not a realistic competitive advantage for your business.
I would conclude by saying that you should minimize the diversity of cloud products as much as is reasonable for your operations. Keeping AWS to just EC2, EBS, Route53, etc., ensures that any need to migrate to a different cloud provider is actually tenable. Put the intelligence into your application/product, not the plumbing around it.
It depends. If you're just another Etsy store, or a SaaS that integrates with other SaaSes, then yes. The entire internet will be down if AWS is down and nobody will blame you.
But if your customers are using your app on a job site and they can't work without it, you'll lose everyone if you have a single outage of even a few minutes.
I have a client in this space and they've handled this by making the app work offline and by pushing a few high-value things outside of AWS to survive an outage of their main stack, such as access grants, and the app can start to use MMS to send its data by including a token with the photo and job info and then this gets synced later by actually adding the token receiver to a user list as if you'd added them through the main UI.
Speaking as someone involved in cost reductions in cloud services, there’s definitely a push towards taming the large bills. However hardly anyone is planning to move for EKS/RDS/GKE to self hosted OpenShift/OKD clusters.
Also, setting big data compute infrastructure is a bit different than traditional server infrastructure, if one has never done it before. Your compute, storage, and network infrastructure are critical components and need to be designed and tuned together. I know. We did it. And now some 7 years later we've moved it all to the cloud.
We've hardly even begun the meaningfully productive AI side of cloud services for business, and that's going to be gigantic. TechCrunch is doing the equivalent of declaring the backlash against the Internet has begun, circa 2001.
Maybe it's time businesses actually did some honest assessment on cloud providers and their services?
So many managers appear to think the cloud is powered by magic pixie dust.
Even I, as an ordinary user, won’t put my data without strong client side encryption in the cloud (even then there is a risk). How come major international (say, European) companies store their data in servers of the American countries? What prevents US intelligence agencies and even corporate from snooping into valuable business data in order to stay ahead?
It’s like I back up my data to my competitor’s computer. I think there was a decade where the situation was not as clear as it’s now, and cloud companies could collect a lot of data. I don’t think it’s that easy anymore, with the spread of client side encryption and so on (although e2ee has limitations too).
Now if you want to get your tinfoil hat on and say "If it's server-side, then what's stopping Amazon from just lying", the answer is independent auditors.
You might not be able to get a tour of the inner workings of these services, but multi-billion dollar customers absolutely ask for it and get it under NDA's.
Also, client side encryption works for things like backups. End to end encryption rules out a lot of features. Therefore lot of companies store data in plaintext (data base, web servers, user facing services etc).
That being said I think what we will see is a move from propriety tools and products to essentially cloud VMs. No one really wants to run their own hardware but likewise no one wants to pay for SQS, DynamoDB, SES, API Gateway etc (insert your preferred cloud provider’s proprietary product)
I wrote a "Mist Computing" manifesto in 2015 while working at Google, capturing similar concerns at that time:
https://docs.google.com/document/d/1SYiDABLgOMljsWl2SDrz5jRL...
Growing companies that go off cloud prematurely will quickly relearn the forgotten lessons of why the cloud was built in the first place.
https://www.db.com/news/detail/20201204-deutsche-bank-and-go...
https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...
Also, from experience, going with a "smaller" cloud provider and opting for a dedicated machine can already reduce the bill by a huge amount.
The position of power of the big 3 - AWS, GCP and Azure* - gives them almost 66% of the market - that doesn't seem right.
https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne...
Even Hetzner maintains a few repos for that:
https://github.com/orgs/hetznercloud/repositories?q=kubernet...
Sure, hybrid approaches do work; keep your main infra in the cloud and supplement it with some cattle rented servers (that can fail with no prod impact) to handle predictable workloads.
But think it very thoroughly before getting fully back to “on my premises”. You’ll discover a world that you never knew it existed.