I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another poster commented, I wouldn't be surprised if it's taken down at some point.
If your business invests in physical servers anticipating strong growth next year then later finds out actually we're going into a recession and those servers are no longer needed, then that's a sunk cost.
With cloud if demand drops you can scale up and down as needed. Helping customers cut costs during difficult times makes sense since those customers are more likely to survive and stay with you through good times.
So in context I think this article makes sense since long-term sustainable growth of AWS should be linked with the growth of their customers' businesses.
Cloud vendors also mostly sell minimum use packages for discounts in the range of 20 to 80% (called e.g. "committed use discount" or "compute savings plan"). Lots of businesses use those, because two-digit discounts are real money, but they might find themselves in the same spot as with physical hardware they don't need...
And cloud proponents pretend data center / rack space / server leasing doesn't exist either, for those trying to avoid large up front costs.
It also means some poor fuck at AWS gets woken up in the middle of the night instead of me when things go to shit.
It absolutely comes at a cost, and might not be the right fit for an organisation that's absolutely on top of it's hardware requirements and can afford to divert resources from new development work. For the rest of us it saves a lot of dev hours that would have otherwise been spent in pointless meetings or debating the best implementation of whatever half-baked stack has oozed it's way out of the organisation in an attempt to replicate what's handed to you with a cloud solution.
And endless orgies of "call for pricing" with hardware vendors and hosting. Shitty websites where you can buy preconfigured servers somewhat cheaply, or vendor websites where you can configure everything but overpay. Useless sales-droids trying to "value-add" stuff on top.
Cloud buys are a lot friendlier, because you only have the one cloud vendor to worry about. Entry level you just pay list price by clicking a button. If you buy a lot, you are big enough to have your own business people to hammer out a rebate on list price, still very easy, still very simple. But overall still more expensive unfortunately.
I'd hope there aren't actually hours of meetings for a single $5/mo VM?
But I would hope there are reviews and meetings when deploying enough of these to amount to real money. Companies that don't do that soon enough find themselves with a million dollar AWS bill without understanding what's going on.
Spend is spend, it's vital to understand what is being spent on what and why.
Slightly exaggerated in the case of the $5 machine, probably 2-3 manhours total but it took 4 days for it to be deployed instead of ~5 minutes. We did spent tens of hours justifying why the business should spend ~$100 more per month on a production system where the metrics clearly indicated that it was resource constrained.
The same IT department that demanded we justify every penny spent did not apply any of that rigour to their own spending. Control over the deployment of resources was used as a political tool to increase their headcount.
> I would hope there are reviews and meetings when deploying enough of these to amount to real money. Companies that don't do that soon enough find themselves with a million dollar AWS bill without understanding what's going on.
I consider the judicious use of resources to be part of my job as a software engineer. A development team that isn't considering how they can reduce spend, tidy up, or right-size their resources is a massive red flag to me. Organisations frequently shoot themselves in the foot by shifting that responsibility away from the development team. The result is usually factional infighting and more meetings.
Also it's going to be simpler to provision your base (commited use) on the cloud and then handle bursts on the cloud, than it is to have your base on prem and burst to the cloud.
You can buy physical servers in leasing ,turning it into opex
You can also rent them for little bit extra via managed dedicated servers from vendors like OVH.
Not going with a big cloud provider def doesn’t mean that you need to buy physical servers and build an on-prem data center.
...while forgetting to have sane on-call rotation for cloud you also need at least 3 people on that rotation that are also clued in on cloud operation enough. Sure they can be "developers" but if your app architecture requires so little maintenance and flea removal that they are not doing ops jobs much, chances are so would it in either rented or dedicated server env.
Yes, but that sunk cost is probably still lower than what you paid AWS for the option to scale up and down.
A really cheap server leasing deal will cost you yearly about as much as the purchase price of the server. With opaque AWS services it is probably more like a month of subscription to pay for the hardware that you are indirectly using.
They were entirely unusable.
Opening a relatively small file in notepad could take multiple minutes. OS click and typing response times were measured in seconds.
Despite wasting thousands of developer hours each year, they refused to upgrade their data center. Probably because doing so would have been a major budget fight that requires an executive to actually advocate for something instead of making their characteristic animalistic grunts of agreement.
For better or worse I haven't seen the same issue with cloud expenditure. It seems to be perceived as a necessary expense, rather than the engineering department getting ideas above their station.
Typing/executing in Powershell was just as slow
Thankfully after they understood the problem it only took 8 months of procurement, techs going to the data center 10+ times with endless screw ups, and everyone pointing the finger at each other.
While the cloud sucks in many ways the traditional setup has big problems as soon as you hit a midsize company ime.
A cloud vendor (who will be nameless as I signed an NDA specifically that prevents me from disparaging them; but one of the big three) ran out of capacity for me and it was 3 months before they managed to fix it. -- that was with a couple million a month in spend.
Cloud is still servers; you just depend on someone elses capacity management skills and you hope that there isn't a rush to populate a location (like when a region goes down and everyone's auto-provisioners move regions to yours)
I have to deal with a grumpy finance guy that thinks my whole department is overpaid already, especially so if we might use the dreaded `CapEx` word.
Once you have on-premises you need people that know switches, routers, rackmount server, hardware, virtualization, etc, plus keeping all of that properly maintained (security patches, IaC, periodic updates, analyzing performance, making sure it's properly architected, etc).
I often see people saying it's the same cost or less but it's really not. Unless you have no idea what you should be doing.
Virtualization, IaC, analyzing performance, right architecture etc is all for later, when you've grown enough to need that.
Yeah, I think it might be a different perspective about when that all should be done.
I tend to do that right from the beginning because I often see it snowball later on and nobody ever fixes it or does it "properly" (in my opinion, possibly not the right one).
But that's a good point, no doubt.
If a business invest into a cloud infrastructure and create a binding contract for 5 years, only to find out that they actually want to abandon that project a year later, that is also a sunk cost. Long term contracts tend to be cheaper, so its a trade off between saving money vs risk.
It all depend on the risk analysis, how risk averse one want to be, and the economics/liquidity needs.
You don't need to go all in on buying racks and other hardware, when you can rent servers at Hetzner at a cheaper cost than aws.
Really though, it seems like a hybrid on-prem/cloud approach is one to consider. Software like Anthos eases this, though there are also pitfalls with this approach too.
Although in this specific case, being a team at AWS, they are using their own company data centers, so it's essentially on-prem to them.
its quite simple, if workload x can be done 100% cheaper on-prem then its an obvious move (probably) if AWS manage to get that closer to 30-40% then the operational benefits of using AWS make more sense, more workloads, more total spend.
(They finally delivered 3.10 last month at least)
1. https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-a...
> Lambda also optimizes the image and caches it close to where the functions runs so cold start times are the same as for .zip archives.[0]
This[1] article shows almost no discernable difference in .NET cold start times between containerised and regular lambdas.
It's easy to imagine developers pushing up bloated images, slowing startup down and blaming docker/AWS for it.
[0] https://aws.amazon.com/blogs/compute/working-with-lambda-lay...
[1] https://www.kloia.com/blog/aws-lambda-container-image-.net-b...
I'm not so sure it's a black/white true/false. Depends on what goes in the docker image. It's something like for larger deployments docker is faster but for small deployments it's the other way.
And then charging them to use AWS anywhere and outpost!
Why? Using the model they switched to (which uses a different set of AWS services) instead of the model they switched from is a recommendation that the AWS tech advisers that are made available to enterprise customers will make for certain workloads.
Now when they do that, they can also point to this article as additional backing.
AWS doesn't have an equally distributed interest in selling all of its products. Some AWS products exist because customers need/demand them and others exist because they provide higher margins and tighter lock-in to Amazon: the first type of products are great for customer acquisition, the role of their sales folk is to then convince people using the former to migrate to the latter.
I've done the AWS solutions architect associate level cert and I can tell you first-hand experience that in order to pass the exam you need to memorize a lot of AWS propaganda that was written primarily to optimize AWS profit, not to optimize customer satisfaction. How many of those solution architects take those materials with a grain of salt vs how many of them genuinely believe that crap, I don't know.
I'm still trying to figure out which one Aurora and Cognito fall under.
> Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.
I got a bit of a chuckle out of this. There's no way to move from Aurora to RDS short of... 2 minutes of actual work and a lot of waiting around due to the limitations of the hardware?
I get that it's not as easy as a literal button click, but this isn't vendor lock in.
Physical Replication should be something any database can offer given its something cross database migrations used decades ago with no problem.
The sarcasm was warranted.
If the CEO asked how long a migration will take would you respond "2 minutes of engineering time"?
Obviously I'm going to spend a lot more time communicating the details of the situation to the CEO of a company that is paying me, than I'm going to spend communicating in a Hacker News comment. But as it turns out, no amount of communication is going to be effective if people don't bother to read past the first opportunity they see to jump in with a correction, even if that means stopping reading mid-sentence.
I'm not sure what I was supposed to take away from your cryptic sentence. Is there a two minute solution to this problem that you are smugly keeping to yourself so you can mock people replying to you?
"There's no way to move from Aurora to RDS short of... 2 minutes of actual work and a lot of waiting around due to the limitations of the hardware?"
You seem to be having trouble getting past the word "and", so I've helpfully italicized the part you've repeatedly missed or ignored.
Now sure, that's a bit vague, but if you want more details it might have been advisable to ask a question rather than simply ignoring half the sentence because you don't understand it and jumping in with a correction.
And honestly, even if it's vague on some details, there's no universe in which "2 minutes and a lot of waiting around" = "2 minutes". Whatever vagueness you might accuse me of, that fact isn't vague.
that is fair answer! AFAIK, there are two things that consider:
* Aurora do have some vendor-locking feature if I'm not wrong?
* moving from Aurora to PostGres will lead to downtime of 2 minutes + unknown waiting, where this is not a case when you convert from PostGres to Aurora.
Consider: is that impracticality caused by Amazon creating vendor lock-in? Or is that impracticality caused by the fact that reading terabytes of data from storage, transferring it over the network, and writing it into storage is inherently slow because of the physical limitations of hardware, no matter what vendor you're using?
It's a bit odd for me to be in the position of defending Amazon here. I genuinely don't like them, don't use them, and generally do think they're guilty of creating a lot of vendor lock-in. But this is legitimately not an example of any of that.
As is, I've no responsibility to show you anything, and you're just making unwarranted assumptions about what I have and haven't considered, based on a pretty selective reading of what I've said.
Just like easy subscribe-online publications that will have you call during 2 hours with a rep pushing you discounts or whatever to cancel such subscription.
Just not cool
Just I'd try to stay away from amazon if I could
I will also mention that the AWS team we were working with on this didn't mention DMS, and, when directly asked, literally told me there was no easy way to do an Aurora -> RDS migration.
Previously it wasn't great, but since they started using Logical Replication for Postgres, it's gotten far better than it was in the past.
I'm pretty happy with the monolith that we run at our business and this seems to validate our decision to stick to that monolith, but I'm also pretty confident that where we use AWS Lambda, serverless is absolutely the right way to go.
For example, I've written a Lambda application to reply to webhook calls and send API calls whenever those come in. It costs maybe $2 per month to run in compute and requests. Would that make more sense to rewrite as a monolith and run on EC2? I really doubt it.
In this article, they didn't bolt the serverless architecture onto another existing monolith, but rather rewrote the Step Functions and Lambda functions to be a single ECS task.
When I have 200 things that cost $5/mo each to run but fit nicely on a single 8core/32gb ram server.. then this lambda stuff starts to seem crazy expensive right?
I like AWS and I would still recommend it. It saves some work but also creates new stuff to do. Especially if you also want the costs be manageable. Automatic updates, configuring a firewall + reverse proxy with automatic certificate renewals and you favorite deployment mechanism isn't more complicated or labor intensive than managing a small application with AWS. You need to interface it just like software you run on your server.
One of the services I host needed to be authenticated by IP. Happens. You easily get a static IP on AWS for incoming traffic. No problem and cheap too. Now try to get one for the other direction... Possible too, but maintenance just became at least as labor intensive as hosting your own machine. AWS just has to fit your scenario and I think many people overestimate how comparatively easier it has become to host a server with feasible security today. Chances are your databases would be less public than if you skip the AWS documentation.
I think you might be unintentionally arguing with a strawman, as everyone else here is talking about using monoliths instead of that.
Few people want to administer a bunch of micro services themselves, but running a single service on a box is pretty low effort, even if you duplicate it for fail over/redundancy
In software engineering, a monolithic application describes a single-tiered software application in which the user interface and data access code are combined into a single program from a single platform.
"mono" stands for one/alone/singular, so monolithic is kinda defined to be exactly that, yes.
You can still have multiple monoliths, but they wouldn't communicate with each other and would be entirely separate applications.
IMO, distributed software is more practical for working development than for technical reasons.
We all know from basic stuff that performing software comes from single structures that does not require packing and unpacking data But scaling large applications is hard, and it was much more expensive back then. Now that we overreacted to microservices we will overreact to monoliths again. And we will bounce many more times until AI take our jobs and do the loop itself
Why? they're still using "AWS stuff" - EC2 and ECS etc. Serverless is a fraction of the services AWS offers.
AWS actively promote ways of reducing customers bills. This article could be considered a puff piece for the AWS Compute Savings Plan:
At some point the industry will wake up to the fact the AWS pricing pages are the real API docs, meanwhile dumb shit like this will keep happening over and over again, and AWS absolutely are not to blame for it, any more than e.g. a vendor of cabling is guilty of burning down the house of someone who plugged 10 electric heaters into a chain of double-gang power extension cords
Not at all. My time working with AWS reps, they never pushed a particular way of doing things. Rather, they tried to make what we wanted to do easier. And the caveat was always to test and make decisions on what was important to us. This isn't an anti-AWS article. Rather, it's exactly the type of thing I'd expect from them. Use the right tool for the right job.
Tldr build the right thing.
>"AWS sales and support teams continue to spend much of their time helping customers optimize AWS spend so they can weather this uncertain economy," Brian Olsavsky, Amazon's finance chief, said on a conference call with analysts.[0]
Amazon isn't afraid of this trend, they're embracing it. Better to cannibalise yourself than be disrupted by someone else
https://twitter.com/DanRose999/status/1287944667414196225?s=...
[0] https://www.cnbc.com/2023/04/27/aws-q1-earnings-report-2023....
The key is to look down on nothing, become competent with multiple architects and know which ones not to implement in a use case if the one to use isn’t clear right away
Yes agreed there were some funny business like not selling Chromecast, but the guiding principle was generally to make things customers want...
They most definitely want to as that would most likely mean more money (and promotions) is flowing there.