Hacker News Reader

Comment by epberry | Hacker News Reader

Surprised to see cloud computing will be affected, but then again some analysts reported that AWS growth slowed to only 20% at the end of Q4. But based on what we are seeing they are still having a lot of success with on-prem to cloud migration deals (Southwest Airlines being the most recently announced one).

We've also been tracking the shift in cloud spending patterns that Amazon is describing on earnings calls and how that may be compressing margins. In Q4 this was particularly noticeable, https://www.vantage.sh/cloud-cost-report/2022-q4

View on HN

We are a medium large company (close to 10k employees globally) and the main focus of this year of the data teams is to figure out every possibility to reduce Google Cloud costs.

The savings could be in 7 digits TBH annually. There are so many low hanging fruits no one cares when the day is good. Tables that have no partition expiration that grow into hundreds or even thousands of TBs, buckets with same amount of data that no one looks at, VMs that no service uses, large clusters well above their load (e.g. only 20% used maximum -- and a full replica too), very bad ETL pipelines that cost hundreds of bucks EVERY RUN, very costly queries generated by Looker dashboards EVERY view...there are just too many.

I think FAANG centric folks underestimate how many fairly big orgs are in the same situation.

Big top down push to move to public cloud, basically no thought to operating expenses.. and now theres a ton of low hanging fruit. Most of these same firms have very little ability to measure their costs in a timely, granular manner, and so put in only very blunt processes to try to limit costs.

Last org I was at could probably re-allocate 10% of engineering staff using the cloud to a SWAT team finding cloud cost saves for 2 years before they ran out of stuff that costs 5x their salary to get fix/optimize/kill in the cloud.

A lot of the same orgs CTOs sold themselves on the idea that they'd need less infra people in the cloud. This really has not been the case, and more often the cloud infra people are more expensive too. Meanwhile many need to permanently maintain an on-prem footprint for latency/legal/legacy/etc reasons.. so you end up with net more staff.

I know a handful of Wall St firms that are moving a lot of "big data" type use cases back out of the cloud because its hideously expensive for their use cases.

I have been saying this for years - hybrid cloud has always been the way to go. Put your highly variable workloads in the cloud and autoscale. And anything super GPU or compute hungry that are regular workloads should be run yourself.

Bonus points if you can autoscale those same on prem workloads into the cloud provider of your choice seamlessly

It's the way to go but the battle you have to fight is maintaining internal competency to host the infra. Especially if you have any kind of security or regulatory overlay its going to mean almost full duplication of a giant stack of processes that parallel each other in cloud / on prem. I think the key as you have outlined is to break out the simple batch compute and CPU/memory intensive or data volume heavy tasks as these are what generate the cloud bills. In general these aren't the focus of your policy constraints, especially if you can claim the data is not "at rest" within these systems.

It pains me to tell you this, but "simple batch compute and CPU/memory intensive or data volume heavy tasks " were the first ones my firm has, unsuccessfully, tried to move to the cloud, no joke.

I think that is probably the right way to go. We have a put a few things on AWS which really don't fit there too well, all on the benefits of shutting down our on-prem estate, 5 years later we still have an on-prem estate.

Ah yes. Ours was to be a 2 year migration tour. Then after 2 years, they said 5 more years. I guess we will see what it looks like in 2030. I can tell you 99% of usage is still on-prem today.

I’m sure Gilligan’s Island In The Cloud would have an audience.

Yeah also sometimes physical hardware is a constraint on waste. Quality & performance of code often has a way of degrading to the point it saturates the available hardware.

In the absence of friction to buy more immediate incremental capacity, teams may never optimize their code..

This may be good when you are "moving fast & breaking things".

This may be bad when you are trying to operate an already-scaled business profitably.

From my experience I think maybe internal reporting fits self hosting. You don't need to scale too much, it can tolerate some fails, etc. But I guess it's still complicated when you throw a wrench called BI into the machinery.

The thing is that self hosting kind of already has cost gates. Whenever I needed to buy another $100K of servers, we had to make a deck and have a skip level meeting 2-3 levels up to explain the capacity planning spend.

On the other hand, in AWS, at my last shop.. we were just .. doing stuff. We at one point noticed a non-production RDS instance was going to run us $60k/year.

This was for some small part of our processing, that in on-prem land would just occupy a slice of one of our $30k servers. But in cloud, we had the pleasure of $5k/month until we noticed & turned it off.

At the scale our estate worked, we had easily 100 entities that could trigger this kind of spend.

well, its getting into the rearview mirror now, but spectre/meltdown-type vulnerabilities are a big motivator for leaving on-prem. even if you later leave the cloud for on-prem, the period without any phys infra means you can start on a clean slate with better equipment + better DC rental contracts.

oh and it really sucks when your brand new SSD's arrive at your main DC, after a 4 month wait, you open the box and the boxes are empty - someone robbed ya. Those expenses are only linked to the infra team, but the effects reverberate through the engineering org.

I wonder if we will also see a dropoff in attendance for big tech conferences like reInvent this year.

Given how many devs were needlessly getting sent at the shops I worked (think 20% of dev team), yes...

We did the same thing (very small company, but outsized cloud hosting costs due to the nature of our business), and frankly the biggest cost savings for us was shifting some resources off AWS and to other providers.

For us, Cloudfront was a massive expense that is without exagerration an order of magnitude more expensive than other providers, and even in the world of AWS custom pricing, it's a struggle to get cloudfront costs down to even the retail offering of other providers.

FinOps is definitely becoming a pretty hot topic. And, yeah, a lot of companies are realizing that they have some pretty low hanging fruit where they can cut spend significantly.

I've wondered how much of the cloud spend in the world is just junk like this.

I don't mean that cynically or sarcastically. The cloud makes it legitimately more easy to deploy resources; I've experienced this in my own teams. But the flip side is that it is easier to deploy things without thinking about the costs, or deploy things and forget about them, and management can just shrug and say "yes, it's cloud, it must be cheaper and better".

This is by no means a cloud-unique problem. Watch any datacenter a company has been at for a few years get shut down and moved to the cloud. The stream of "hey, who owns this?" and "are we still using this?" and "I just yanked the network cable, has anyone noticed?" is morbidly amusing. Even if you think you're tracking everything, you quite likely aren't, because the owners don't update their own tracking info very well.

But cloud makes this worse.

So I just find myself wondering how much of AWS profit is on this sort of stuff.

I myself have S3 buckets full of crap. As I do watch my spend, said buckets full of crap are in the ~10-50$/month range. At that price point it is difficult to justify engineering time to clean out the crap. But it's not hard to imagine that scaling up by three or more factors of magnitude.

We saw this one iteration back with VMWare. Once it became easy to spin up a VM without getting a PO for a server and finding rack space, the number of "servers" in the org exploded. And of course we lost track of what they all did, and who they belonged to... but there was actually a healthy ecosystem of tools built for (and by) VMWare to manage this situation. The cloud providers seem incentivised to make tracking real costs as difficult as possible. At least with VMWare once you paid for it, it was yours.

There’s great tooling on cloud providers for tracking costs. The pain point is that if you never set it up it’s incredibly difficult and time consuming to add it after the fact through tagging.

Yeah exactly. I feel that recklessly opening new projects and then abandoned them, or refactored multiple times with different teams, quickly added up to the number of rss no one owns.

Last couple of weeks the whole data org underwent a forced upgrade of certain components on Google cloud and we got quite of clusters no one knows who owns.

I don't know but I'm itchy to start finding ways to build new projects because that is the best way for techy people to grab ownership and promition.

Feels like there is a startup in there? Give it your account number, and read only access keys, and it tells you what and where and how to change things.

Lots of players in this space. At my work we have:

https://www.apptio.com/products/cloudability/

Just curious what do you think about the service? Is it a lot deeper than what Google itself provides? We are just using Google own service to turn down costs.

So Apptio is a larger application for finance people, with some cloud cost management built in. They are the main users, I just have read only access an login perhaps monthly to view budgets and cloud spend.

I just have access to small bits and pieces, but what I have seen I like.

Thanks for sharing!

Just a note that Google itself is doing that so one has to better than that. I think the ideal candidate is a senior data architect/ops person who has xtensive hand on experience. Sadly I'm so far from it.

Sigh, I'm late for everything in my life.

FWIW the features that GCP exposes for this is way better/easier than AWS's equivalent. So much better

I never used AWS (and am just a peasant using GCP so don't know everything). What differences did you find to be extremely helpful?

GCP puts automatic analysis recommendations in the UI right next to the resources in the table of VM/etc names, and IMO the recommendations (for both upsizing and downsizing) are dramatically more accurate than what AWS gives you

Disparaging yourself is a good way to make sure you don't achieve anything. You should stop.

There are companies out there. I expect to see a number of them when I'm at Kubecon in a month or so.

Yeah. I can think about consultants making $$ giving full analysis of cloud costs and ways to reduce them. It's actually not very technically challenging TBH. All those low hanging fruits can be grabbed by a junior enginner or even a non tech senior person who is a deep user of such services.

Start by owning the datacenter. You might just as well have the data that no one looks at and the 20% load clusters in one, two, or a few 48U racks [1] in the room next to you. Keep whatever is latency-sensitive still on someone else's computer, i.e. the cloud.

[1] Get a "server as they should be", https://oxide.computer (no affiliation)

> Start by owning the datacenter

But then, when uncontrolled inefficiencies need to gobble up more hardware, you would have to buy, install, network, maintain, and understand this new hardware.

Unused resources do not gobble up real hardware.

Real inefficiencies do, but then you will have the more detailed data the GP is talking about.

I really want to learn in an on prem env (and learning on my own 32GB 2TB x-handed server is probably just scratching the surface). Sadly I'm a junior so could not make that call. I also have no idea how much cost the transfer to on prem will be.

No time like today.

You don't need to make a large datacenter, or to migrate everything at once. Find some low-risk services, get some consultant help you improve some room on your premises (you'll need to care about security, electricity and cooling), get a pair of network contracts and one hack of servers (or maybe even 4 servers), and move those services. Favor things that are used internally.

If you are looking for 7 figures of savings just by picking the easy fruit, you will almost certainly get positive ROI on that. Just do not move critical services if you don't know what you are doing.

I work for a tiny startup and while things are good, we are prioritizing reducing cloud costs as well. Our bill is relatively tiny right now, but as we grow it’ll become a bit more unsightly. Best to do it now and get that over with!

At a tiny startup I'd be more concerned about selling product/delivering to get the next round and keep the company running than saving few 100$ on infra cost.

Not every startup is funded by VC vultures that set aggressive timelines

Deadlines don't matter - if you're in a startup you're trying to test the market fit and iterateing to corner the market. Worrying about micro optimizing infra cost and talking about scaling in this context in my opinion is premature optimization - root of all evil.

This, and cloud security, have become my niche at my small company. Keeping these things tidy now will save us so much headache later. It's a bit of a battle fighting product for time but these are hills worth dying on imo.

Interesting. I wonder how one learns it. I guess I first need to jump to a full Cloud admin position to learn on job.

I don't really have a secret. I just have 8+ years experience with one cloud provider across 4 different companies as an SDE, so I've seen all the highs and lows that come with using it. Current company keeps bugging me to take some certs but doesn't seem worth the effort..

Thanks, I guess I just need more experience. But first let me move to a real cloud admin position...

Yep, we're in the same boat. About 4mo ago we started looking more closely at AWS costs. Tons of low-hanging fruit.

Yep I’m in the same size company and we’re doing work to reduce GCP costs. You are so right, our team did a sprint or two of work recently and shaved $100k off our monthly bill which in our company is small potatoes

just messy bread making machines, as joel spoelsky would likely put it

https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...

cloud computings darker secret is armies of overworked sysadmins and SRE's toiling in obscurity through exhausting oncall hours behind the scenes to keep everything running despite these sorts of cuts and mountains of tech debt.. anyone whos worked for a hosting provider can attest.

Is that really a secret? I have worked both on the inside at AWS, and on the outside at groups which used AWS and Google Cloud. “They will do the huge amount of sysadmin toil” is, like, the number one selling point versus using on-prem systems, and its what justifies the prices.

The cloud providers like to pretend their secret sauce is massive scale automation. They have tooling, alerting, etc. to make this all work with no people at all. Of course it's all BS, and the people I know who came from AWS were some of the worst for throwing bodies at problems and manually tweaking shit without trying to understand or automate solutions.

It is their secret sauce and like a startup who licks and sticks envelopes initially while proving their business model, eventually it is automated.

But the thirst for growth is never quenched, so there's always new features, new services. And maintenance of that automation. The work never ends.

There's a 25-50% operations tax every AWS employee pays. But they're handling the operations load that several employees were 10y ago. I used to wake up in the middle of the night to restart individual machines. Now I'm alerted for much much more complex problems. But i'm still woken up :(

They throw bodies because they have them and they are looking to solve a problem.

And they do automate, claims that a service like S3 does not have automation are totally ridiculous, and few other general hosting providers have AWS scale here.

That said, their development pace is crazy, with plenty of partially built solutions -> so yes, until something scales enough to need it I'm not surprised they have sysadmin toil. Reality is even the VMware worlds and other onprem stuff, even when "automated" have plenty of turmoil just staying fresh / current / backed up / secured etc. So that's what AWS is selling.

I admin a small part of our Google cloud part time and I feel I absolutely have no time and no idea how to drill deeper than clickOps. Knowing where is everything and the general risks of toggling a switch are already nerve breaking.

Sometimes I wish I could learn everything from scratch. I wish I could join an on prem team to learn all those small details slowly and painfully.

> I wish I could join an on prem team to learn all those small details slowly and painfully.

Learning "from the middle" of a tech stack outwards is the default case for everyone, so don't be so hard on yourself. There is no "from scratch", never was even in the Bombe days, just a question of how many abstraction layers you want to tackle. Give yourself time to learn and plenty of room to fail and try again, and "all those small details" will ingrain themselves into you as second nature.

All of big tech is throwing bodies at problems at this point since decades of data is all rotted and programmatic solutions simply don't work. IF you stay at Amazon after all of this attrition, your life will be hell.

And yet AWS is pretty reliable overall, and certainly better than most in-house systems even at large companies.

I know multiple people who have worked on AWS. There is definitely some manual stuff going on, but there is also considerable infrastructure and automation in other areas.

But it's not BS, automation is essential.

I'd look for other reasons..

The amount of sysadmin/SRE toil in some parts of the stack (all the distributed computing coordination layers) is super-linear in the growth of the system, but the nuanced version of the claim is that they can make up for it in the other parts of the stack (hardware cost and better global availability). A lot of people who have never seen these systems from the inside would be surprised at (1) how much engineering work is put into just getting things like "Postgres but it scales" and "Kubernetes but it works on our proprietary VM manager" to work and (2) how many internal incidents these cloud providers have that are quietly managed before they rise to the level where a customer would notice or care.

(opinions my own) - I actually do work for $FAANG and can confirm this statement. Mountains of tech debt, 20+ year old tech running most of the stack, most tooling written in dynamic interpreted languages, and above all else every single engineer is expected to be part of on-call rotations, even if your role is not related to infrastructure or operations whatsoever. Even if you don't touch the service(s) your team maintains because you're in a FEE role.

The SDE role at $FAANG I work at is essentially glorified tech support, which makes the leetcode interviewing process so much more ridiculous than it would be by default if you were actually writing code and solving meaningful problems most of the time.

> The SDE role at (FAANG)

I don't think you're generalising to all FAANG, just whichever one you work at.

That said, this has not been my experience at all at another FAANG which doesn't use the TLA "SDE".

I’ve worked at Twitter, Uber, Amazon, and Facebook. They all had mountains of tech debt and oncall burden.

Which one do you work at so I can go there?

I think you're going to find this at nearly all software companies, FAANG or smaller. The sad state of our industry is that everything is so poorly written that it's falling apart daily, and each company has an army of people trying to toss water on the last fire that popped up. This is the natural steady state of an industry that prioritizes schedule and feature cram over quality and craftsmanship, and prioritizes human heroics/toil over process and automation. Move fast and break things.

The key I have found is to have hobbies that take you out of cell reception.

Or at least appear to have those hobbies if you do not.....

The thing about FAANG is that very few experiences can be generalized, even within a single FAANG company. These companies are so massive that culture, tech, etc. can have huge amounts of variance between orgs let alone companies.

Using (FAANG) for anonymity, but you're absolutely correct

It's well known in the ISP industry that being an entry to mid level sysadmin/network eng at a hosting provider that has large numbers of very-low-dollar-amount per month customers is a new circle of personal hell. Working endless tickets from people who pay $20 a month for some VPS. The only way to get out of it is to become senior enough to not care anymore or work for a hosting operator that specializes in only higher dollar value colo/dedicated server customers.

From multiple reliable sources I've heard that AWS's DynanamoDB team/ops is a meat grinder. The open tickets run into thousands with no hope of anyone getting time to work on permanent fixes.

The “best” thing about layoffs/attrition is that onerous task you used to split with several people you now have to do twice as often.

What's the "dark secret" exactly? The entire point of AWS is companies being able to outsource this army of sysadmins so they don't have to hire them themselves. That's also why they pay 3-5x the cost of buying an equivalent server to Amazon.

This is a "you can write bad code in any language" paradigm. Doing computing efficiently takes skill, whether one does it in the cloud or on prem. Cloud solved a few common pain points and, at times of startup / tech boom, had a meteoric rise.

But now many of the companies that moved to the cloud are hitting different challenges and are stuck in a setup that is pretty expensive to maintain. I am not expecting the death of the cloud computing, but that pendulum will likely swing back a little as the companies experiment with other options. I think a company that can help medium sized businesses optimize their compute setup with a mix of on prem and multi-provider cloud will do well. My 2c.

>This is a "you can write bad code in any language" paradigm.

It's extremely true! The fun is that bad code looks totally different from each other. The "bad" Scala type astronaut code would be unrecognizable from Python raw dictionary enthusiast code.

Cloud computing is a massive rip off. From my experience with start ups, they were using things like AWS, because otherwise the investors would be reluctant to put their money in. As in they were afraid of dedicated hosting and something that small business wouldn't be able to handle.

So for instance company would have their hosting costs upped from £500 a month (dedicated servers) to something like £5000, so that they could "scale up", while their dedicated servers could have easily served almost any traffic thrown at them and be more responsive. They have actually never reached the level of traffic that this old set up wouldn't be able to handle, but investors were always saying "what if we get sudden surge of users" which has never materialised. But then it was their money so nobody objected.

I can see that now that hosting costs in the cloud have become ridiculous, cash strapped smaller businesses will be looking for savings and the cloud is quite an obvious area to look for cuts.

> while their dedicated servers could have easily served almost any traffic thrown at them and be more responsive

Sounds great, for some use cases. Particularly if you don't need multiple 9's of uptime, and just need a few 8's. Most companies don't have: * Guaranteed sla's for replacing components when the RAM needs to be reseated, a drive fails, or the fans get too dusty and things overheat

* Redundancy set up in case the network connection fails

* Redundancy set up in case the power fails. (Many places will have a UPS for under an hour, but generators, along with a regular maintenance schedule and processes for ensuring fuel is another thing)

* On site security, to make sure nobody breaks in and does some corporate espionage

* Ultra-high-speed links to the internet

And if the little startup takes on a client with important certification requirements, it can be very helpful to have a cloud provider. A lot of contracts I've taken on over the years ask tough questions during the sales process about our practices regarding the above things and much more.

I have some dedicated servers with over 7 years uptime now. Only time they were unreachable (not down), when datacentre had a network maintenance and that was an extremely rare and short event.

The points you have listed are important, but are really non-issue unless you are co-locating servers in the utility room somewhere, rather than in a proper data centre.

Have you never used colos? They literally tick all the boxes and it would be one of the worst colos I have ever seen to not have 3x9’s.

Cloud computing can cost 3-5x self-managing with the open source versions of the standardized cloud tools that are sold to us.

One big difference working against the cloud is the same thing that worked in it's favour 15 years ago. We don't always remember why we just adopted the cloud. Linux networking at times had scaling issues, and virtualization and linux was not where it is now.

It's astounding how much more a $5 VPS can do. Imagine the medium end. Self-managed hosting tools have come so far to handle production quality, it doesn't even feel fair that it is not more well known. Whether it's Proxmox, or others, it is pretty close to most things out there. Making a tool cloud compatible, but cloud agnostic is one way to avoid a lot of technical debt (assuming it's not created another way).

It doesn't have to be for everyone, but demand is probably slowing in part because of this. In a recession cost efficiency is everything.

I wish Rancher had never stopped the development of 1.6. It was such a simple and beautiful tool to run services.

Now they moved to Kubernetes, it has added a ton of unnecessary complexity. It takes magnitude more effort to run anything on it and they call it progress...

It’s probably making a developers life easier somewhere instead of the users.

Sometimes it can be both but not always.

Reply on news.ycombinator.com