I got pwned by my cloud costs
troyhunt.com
troyhunt.com
Always use your own proxy where the egress is well within your free tier, i.e. do this: Azure|Amazon > Hetzner|Linode > Cloudflare
Why?
Because Cloudflare cache is a massively multi-tenant LRU cache and whilst hot files will be cached well (and with Cloudflare Tiered Cache even better - but this itself is a cost) anything else is still going to expose you to some degree of egress cost.
When I exposed AWS to the web I paid $3k per month to AWS. With Cloudflare in front of AWS I paid $300 per month to AWS. With Linode in front of AWS and behind Cloudflare I paid $20 per month to Linode and about $12 per month to AWS.
A Linode, Hetzner instance... or any other dumb cheap web server that comes with a healthy free tier of bandwidth is all you need to set up a simple nginx reverse proxy and have it cache things to disk https://docs.nginx.com/nginx/admin-guide/content-cache/conte...
Or if caching is your biggest priority then Fastly or Akamai will shine too.
But if you're balancing all considerations and want the cheap "good enough" caching with the DDoS protection, free TLS certs, and unmetered (assuming you aren't imgur or something)... then Cloudflare does a great job at being good enough. And for those sharp edges... drop in a proxy of your own, or layer your CDNs.
Fastly comes close on a lot of fronts (and does better at a few things) but unless you are godlike with Varnish scripting it's a lot harder to make it do what you want than Cloudflare.
So I'm not really responding to OP but rather the commenter I replied to. :)
As a CDN it's pretty great though, I have managed very very large properties behind Cloudflare and they have always gone above and beyond for us when big DDoS have came our way.
AWS/Azure > BunnyCDN > Cloudflare?
Or just straight AWS/Azure > Cloudflare?
Why not directly Hetzner|Linode > Cloudflare?
I hate these 'cloud economics' optimizations that people tend to try.
It is possible but highly unlikely. The more likely scenerio is you just continue overpay like a lot of others waiting for the moment. If that moment happens you realize with the sudden popularity your store inventory is sold out so you couldn't profit off of the extra traffic anyhow.
I disabled all shields for their side and still the same thing. Waste of time
Also Azure and Linode, Scaleway Backblaze and others are part of Cloudflare bandwidth alliance [4] so there shouldn't be egress fees between the two.
It is really only AWS which is a problem, you don't need this setup with any other provider.
[1] https://www.ovhcloud.com/en/public-cloud/object-storage/
[2] https://www.linode.com/products/object-storage/
51. Amazon Lightsail
51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services (e.g., proxying network traffic from Services to the public internet or other destinations or excessive data processing through load balancing or content delivery network (CDN) Services as described in the Documentation), and if you do, we may throttle or suspend your data services or suspend your account.
The clause prohibits you from using Lightsail to cheat other services. So, per their example, you couldn’t set up a Lightsail instance as a reverse proxy between the internet and an ELB, to take advantage of Lightsail’s higher transfer quotas and “in-region” traffic from Lightsail to ELBs.
Hosting a site on Lightsail and hosting other things on other AWS services is fine.
(search HN and reddit for that URL, you'll see they've been around and recommended for a really long time).
see https://blog.cloudflare.com/introducing-r2-object-storage/
From the Cloudflare blog, it seems R2 would've handled this exact situation - auto-migration of cloud S3-like-storage objects - download from cloud-storage just once and cache in R2 for Cloudflare to serve.
Would love to find out if you can write to any/every region and have things replicate, or if you have to write to a single region. BunnyCDN's edge storage solution looked interesting until I found out it only supported writes to a single region.
Hoping R2 might be my savior here, otherwise will probably have to roll my own active-active minio cluster, which I'm not looking forward to maintaining. Other suggestions welcome!
Team I'm in at the moment is in the early stages of cloud adoption but the company in total has fell hook line and sinker for AWS. When I mentioned the cost there is always an excuse.
The main one being that you don't have to hire sysadmins anymore as that's taken care now by AWS. Ah yes but they have actually been replaced with a "DevOps" team plus just our department now spend > £1 million per year to AWS in hosting costs. A 20% reduction in those fees could pay for a few sysadmin(s).
The next one is that no other vendor would be able to supply the kit. You know StackOverflow is able to run on a single webserver (https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...). Plus many of the other providers have loads of instances available.
I mean I'm not against cloud it's just not the cheapest option if you choose one of the big 3 providers. I use a company called scaleway (https://www.scaleway.com/en/) they have all the essential cloud services you need and everything else you can run yourself in docker or k8s.
They still serve a lot more traffic than I do and I have hundreds of instances; thousands of containers.
Scale can depend on many things.
Here's a couple of reasons why it can easily be thousands:
1) Cronjobs, CI jobs, ETL, FaaS are all systems that exist. What used to be a process is now a container. (one need only check the PID count on their local machine to know that this can be many quite easily).
2) Microservices; I'm a larger fan of fat "services" but doing actual micro services tends to leave you with a lot of containers running
3) Actual compute need. If my original hosting strategy was thousands of machines, well, I'm going to have thousands of containers, if not more.
i don't remember who said it but a quote i really like is "it's not finished when there's nothing left to add, it's finished when there's nothing left to take away"
A similar “scale” e-commerce site would be significantly more load, have more dynamic data, and just be overall harder to run.
It's basically a form of permanent debt. Faster product market fit, higher long term infrastructure costs until you have enough breathing room to start pulling it into your own datacenter. At that point you have some negotiating leverage with the cloud provider.
On the other hand, if you're not looking for explosive growth man oh man is DigitalOcean or anyone of a number good providers of good old VPSes / Cloud-lite.
I've worked with teams on both sides, and everyone is gonna have to deal with figuring out how to run at scale, it's just different ways of achieving that.
I've worked with teams that manage their own infrastructure with dedicated servers, and not having to think about scaling for a long time as the one beefy server could just take whatever load you threw at it.
I've also worked with teams who don't manage their own infrastructure and thought they were ready to scale without issues, but once the scale actually happened, it turned out there was more things to consider than just the amount of servers you run, race-conditions were everywhere but no one thought about that.
Definitely a case of "right tool for the right job", but I don't think it's as easy as "Self-managed: harder to scale, PaaS/Cloud: easy peazy to scale".
There is definitely a place for AWS/Azure and their offering of services is fantastic but they are not a silver bullet for scaling your website to millions of active user.
On another point though the vast majority of websites you'll ever build won't have that level of active users. It's a good problem to have though as it means your site is doing really well.
This is actually one of the strengths of the cloud, startups that can't afford talent throw compute resources at the problem. Running your own servers isn't hard per se, but it requires a certain breadth of less centrally documented knowledge than the cloud and a willingness to fuss. Developers like that can often command higher prices than most startups pay these days :)
Putting it all on the devs is exactly how you end up in the haveibeenpwned database and on the cover of magazines (for the wrong reasons).
We’ve traded sysadmins for more expensive DevOps. I would love to see a study on if we actually hire less people than if we just did it the old school ways.
Depends on the team size of the said startup [0]. In my opinion, tech-shops are better off using new-age cloud providers like fly.io / glitch.com / render.com / railway.app / replit.com / deno.com / workers.dev etc [1].
[0] https://tailscale.com/blog/modules-monoliths-and-microservic...
For ~100eur/month on hertzner you can get a 16core Zen3, 128GB RAM with 8TB of NVMe SSD.
Unless your stack is horrendously badly optimised you can serve SO MUCH traffic off that - definitely billions of postgres records without breaking a sweat.
So the scale argument somewhat disappears - if anything, people end up adding much more complexity to the product to get round the high hardware costs of the cloud (complex caching systems for example, instead of just throwing loads of hardware at the problem).
What option is that? The closest I see is the CCX41, but that is 40% more expensive, 140 Eur/month, half the RAM (64 GB) and ~4% of the disk space (360 GB)
I guess I shouldn't be surprised, but I do find myself often surprised to realize that for a younger generation of developers they have never experienced hosting on bare metal. So they have not been exposed to costs & benefits vs. the cloud approach and feel that no local machine could ever be as fast as AWS. Even though in reality even a pedestrian server is immensely faster and cheaper than any AWS offering.
Now, sure, there are tradeoffs in ease of scaling up and other considerations, but it's good to keep and eye on the actualy tradeoffs you're making and how much it's costing.
Not only can you end up spending $10k of engineering time to optimize and test a random, non-core-competency bit of code instead of an extra $1k/year on hardware, you also have to maintain the optimized code instead of the simpler code.
Maybe I just worked at companies that did a poor job of managing servers, or had a dysfunctional relationship between software engineering and operations, but at least that's no longer something I have to worry about in a cloud environment. If spending a little extra on hardware is the best solution to the problem, process/planning/politics won't get in the way.
Yes, that's the beauty of it and sometimes you need it.
OTOH, how often do you need to grow capacity without any lead time like that? If you are in a hyper-growth stage in a startup you absolutely need it and it is a lifesaver.
But, most companies never see a hyper-growth stage. Even those which do, it's a relatively short timeframe (you can't grow exponentially very long).
All the rest of the time it's a fairly large premium to pay just in case another hyper-growth period happens. Sometimes it's totally worth it. But good to review the likelyhood and cost tradeoffs every now and then.
That's true with owning your hardware, but what about renting from Hetzner/OVH/etc? You get servers set up in minutes unless you have a very specific request (the only time I've had lead time with these providers is when I had a very custom request, a machine with 300+ TB of storage - yes that is not a typo). Everything else has been delivered pretty much instantly.
But even if let's say you have a very specific use-case such as needing a 300TB server that would typically require lead-time, well in that case the prices are so cheap that you can just keep it around all the time sitting mostly unused and still come out ahead compared to cloud pricing.
It stopped to be the case in pandemic times at most cloud operators due to general hardware and capacity shortage.
We run most workloads on a 3.5GB/2 "vCPU" box. This costs around $70/month per instance. We actually haven't scaled this out past 8 instances, at a cost of $560/month (and that has been extremely rare).
On bare metal we could have ran it on a $100/month 16core/128GB box and always had that capacity in reserve. While app service gives a lot of benefits, the scalability argument is somewhat moot as basically you can provision all the capacity you would scale to 24/7 and still the same/less than cloud.
Maybe it's just the projects I've worked on, but I haven't really ever seen people require 100x or 1000x the capacity in a very short period of time (which obviously bare metal could not do). I've seen traffic grow that much - but generally over weeks, months or years.
Actually AWS won't help you here. I have literally been on a 2 day training course or aurora with AWS and the explanation of how to scale was actually just the same as any traditional non-cloud explanation. Correct usage of indexes, partitioning data, optimising queries (especially any non trivial query output by an ORM) and read replicas.
In terms of explosive growth if you're talking about something like google or tiktok again slapping it all in AWS will not automatically just work. There is a lot of engineering that you'll need to get to their level.
I also think you haven't really looked at the SO link I sent through with thoughtful engineering they have huge user base with a tiny footprint.
> DigitalOcean or anyone of a number good providers of good old VPSes / Cloud-lite
Not sure why you are dunking on DO here they are a fully fledged cloud provider with much the same stuff you would need. You can also run up a huge bill on DO as well.
Without any effort you can stand up a redundant, high availability deployment. With all of the data encrypted at rest. And configure nightly backups, which are stored on redundant storage in multiple physical locations and also encrypted. You can then restore those backups into a working system with the click of a button. Oh, and minor version patches happen automatically with no downtime. And you can click a button to do major version updates.
The last time I did analysis on it, which was a while ago, all of those features cost us less than 8 hours of my time each year. It would probably take more than 8 hours of my time each year just to handle security patches on the systems. Let alone the amount of engineering that it would take to get a system as redundant and reliable as a DB in RDS. I will happily pay them to take all of that off my plate so I can focus on other things, like optimizing the queries.
Yes, it is seductive. Sometimes worth it.
But realize you'll be paying monthly in perpetuity for the convenience of that one-time setup which could've been done a a few days, give or take.
> all of those features cost us less than 8 hours of my time each year
I'm surprised! Our RDS costs are about 10 engineering hours per month (120 eng/hrs per year). This is with hardly any customer traffic or data yet (early startup phase).
It's worth it for now, but it'll become unreasonably expensive later.
Most of the problems here will be DBA problems like understanding query plans and such. Even with AWS RDB, I’ve had to upload various setting files to tweak tunables to get things working.
You can hire a "few" sysadmins for 200k/year?
https://uk.indeed.com/jobs?q=System%20Administrator&vjk=5149...
Probably not at FAANG level salaries but I doubt there are many sysadmins working for FAANG companies anymore.
DevOps btw are more expensive and infact in the UK DevOps can be higher paid that a developer. I suspect most of the DevOps working for this company are on £65k+. According to:
https://ifs.org.uk/tools_and_resources/where_do_you_fit_in
That puts those earners in the top 3% or from that website:
" In the below graph, the alternatively shaded sections represent the different decile groups. As you can see, you are in the 10th decile group.
In conclusion, Your income is so high that you lie beyond the far right hand side of the chart. "
Dealing with hardware failures, hardware vendors, confusing licensing, having to know SKUs, racking new cabinets, swapping hard drives, patching servers - it's all awful work. When you go cloud only, you can be more productive instead of dealing with some of that nonsense work.
If cloud improved QOL for ALL employees I'd agree but I think it just shifts work around and costs more.
I just wonder what happens if a ram or hd failure hits a cloud provider node. Is the architecture on average really able to come over such failures without help and intervention.
By the time you're running a [catering business | massive sysadmin team], you're already a huge success. Congratulations!
And, honestly, I miss the old days. Today, $cloud has some weird spasms where you suddenly get an influx of connection timeouts or tasks waiting for aeons to get scheduled and you just can't log in to a switch or a machine and figure out what the exact hell is going on. You just watch the evergreen $cloud status page, maybe file some tickets and pray someone bothers to investigate, or maybe live with those random hiccups "sorry $boss, everything is 100% good on our side, it's $cloud misbehaving today", adding more resilience -> complexity -> unreliability in the name of reliability to the system. Either way, with the clouds I feel handicapped, lacking the ability to diagnose things when they go wrong.
I don't miss those three days we spent fighting a kernel panic. Was about a decade ago - we outgrew the hardware and had to get a new one with a badass-at-the-time 10GB SFP+ NIC that worked nice for the first few weeks but then its driver suddenly decided to throw some tantrums on almost a hourly basis. I don't even remember the details - a lot of time flew since then, but thankfully we found some patch somewhere in the depths of LKML and the server was a perfect clockwork ever since. That wasn't fun, but that was an one-in-many years incident.
Either way, I do feel that in the ancient ages hardware and software used to be so much more simple and reliable. Like, today people start with those multi-node high-availability all-the-buzzwords Kubernetes-in-the-cloud monstrosities that still fail now and then (because there are so many moving parts shit's just bound to fail at incredible rate), and in the good old days people somehow managed to have a couple of servers in the rack - some proper, some just desktop towers sitting by - and with some duct tape and elbow grease those ran without incidents for years and years.
Have I turned old and sour? Or maybe it's just the nostalgia about the youth, and I've forgotten or diminished most the issues while warmly remembering all the good moments?
The later ppl still do what they did, they just work for Cloud Providers making probably quite a bit more than they did previously.
IMHO its a win win situation for everybody. Less skilled engineers can be “productive” and former sysadmins have huge salaries.
I've met plenty of datacenter technicians that loved they work and the opportunities for growth it provided.
Some companies really know how to manage a datacenter with minimum pain. Some don't.
Each to their own, but I think you'll find there's a fairly significant portion of sysadmins who love that work!
I'm curious about your workload. I tend to only use cloud for workloads where it's either (1) by far the only feasible option (e.g. need GPUs for short periods of time), or else (2) basically free.
> I mean I'm not against cloud it's just not the cheapest option
This is certainly true for most workloads. It's also true that buying is better than renting, but here I am living in a rented apartment.
The logic from on high might be something like "if demand is uncertain and capex is risky, why buy when you can rent?"
I am still waiting for a cloud without these dark patterns. But that will never happen because it‘s leaving a big amount of money on the table by not being hostile.
(edit) looks like that's not the case, I'm sure I used to have to buy a second instance a few years ago if I did want to use more bandwidth that was allocated
For those curious, the overage rate is $10/TB ($0.01/GB) after the transfer included in the plan.
The smallest amount of included transfer is 1TB for the $5/mo VPS.
If you use up your monthly network transfer pool, you can continue to use your Linodes normally. That being said, you will be charged $0.01 for each additional GB at the end of your billing cycle.
Funny enough...Oracle (OCI) makes it better, you can buy oracle"coins" 1to1 with $ and load your account just with what you think you need.
Hard requirement: My image can run on it (freebsd and linux), no proprietary BS, no special stuff, give me vm-"harware" make it fast, make it cheap, make it reliable, that's it..that's it.
And ATM i like oracle hetzner and vultr at most. If one of those change to my disgust i change, no big deal...just some dns rewrite.
This is how mobile and landline phone companies made enormous fortunes before flat rate billing. It’s called post-paid vs pre-paid billing.
"Predatory death-trap pricing" captures the spirit of the thing with rather more clarity. It is wholly intentional after all.
Or craft clothing for goths?
But the "dark" comes from its association with evil: "Defense against the Dark Arts", "The Dark Lord", "Turn to the Dark Side of the Force". It's a clear implication that the people are "selling their souls to the devil": knowingly doing something "a little bit evil" to achieve their aims.
I can see your point- if I'd never seen the term before I might have a similar reaction. But it's quite a common term now I think.
"Mr Politician what do you think about $big_co using dark patterns?"
"Mr Politician what do you think about $big_co engaging in predatory death-trap pricing schemes to defraud consumers?"
What is being asked in one of those questions is clear to everybody. The other is jargon and excludes the majority of the population from understanding the intended meaning. And note, you don't have to agree that it is "predatory death-trap pricing" at all. That is simply in the above sentence an accusation. Words have meanings. That the accusation be understood clearly by as many people as possible is important.
However regarding your other claims- I dont work on websites, I know the term "dark pattern". so your assertion that "nobody else knows what it is" is false. You might then argue that I'm still in tech, but that's just another goalpost that I can reach anyway: if you google for the term you'll find vox articles, explainer sites, even newyorktimes articles using this term.
So yes, it's a common place term and your assertion to the contrary does not hold any water.
As you say they make it hard deliberately.
Edit: Turn out Azure have this:
https://docs.microsoft.com/en-us/azure/cost-management-billi...
CMIIW, it'll be my first cloud provider if I can set one.
Not for actual cloud usage, like an actual pay-as-you-go plan where this would be useful.
https://azure.microsoft.com/en-us/support/legal/offer-detail...
It's very easy to overspend on these big cloud providers
Oh I can't edit the comment now oh well sorry if I've confused anyone.
My own instances go down all the time when I forget to ‘top up’ the account
Contract dispute cases might clarify it, but probably not in the direction any of us is hoping.
But the water company does not actually allow you to install proper taps to regulate the water so you use duct tape to do so, and due to an earthquake something falls on the tap causing your duct tape solution to fail leading to a massive surge of water, leading to your massive water bill.
Did the water company cause this? No. Your duct tape solution wasn’t resilient enough because it didn’t factor in an earthquake. But I would be justifiably mad that my water company does not allow me to install actual taps, and allows unforeseen and unpredictable situations to make me run up huge bills that could otherwise have been avoided with a proper tap.
There are both spending limits and alerting that you could use, but would be impossible to predetermine from Azure's perspective, so they rightly ask you to.
This is very straight forward from their view, before: almost no traffic = almost no costs, now: huge traffic = $$$.
On the other hand, it doesn't seem that Troy did try to talk to them about this and seems to want to eat the costs himself. As it was his mistake. I think that's commendable. I also think with the amount of free advertisement Troy has done for them they'd be open to this and I can imagine we might see a followup post like "MS was so nice they waived my costs".
Exactly this. As a low-level / embedded / non-cloud stuff dev, I've been getting up to speed through all the cloud-ification of the industry, but I'm still scared (not literally ofc) of running most things on my own on any big cloud provider (smaller ones seem more manageable).
I'm reading this and seems like being a customer of cloud services is like walking a dangeous path filled with gotchas and caveats, just jumping from cover to cover while hiding from danger, and hoping you're safe and didn't mess it up so far, "fingers crossed".
Like this tiny detail that he didn't realize was critical, so I would fall on it too plus on another 500s small papercuts: "oh I set cache up, so I hope all is well". "Yeah, no you aren't, I guess you didn't think of this detail about maximum cached file size! Gotcha, Game Over!"
Yeah cloud providers should have clearer communitacions and etc etc... but the fact of today is that they don't. So I'd never sleep well feelin 100% confident that I had covered and taken into account every minuscule detail and possible scenario that could end up being a disaster.
Another advantage is his big network that he can ask for help. There's also a chance that his blog post will reach the right person in Azure and he'll get a reduced bill.
As someone who doesn't have the same network or the "fame", I am concerned about what would have happened to me in that situation.
Maybe it's not a problem when you're dealing with millions of VC money, but there's no way in hell I would host anything in a bandwidth-metered cloud service when my or my own company's money is involved.
Eh, I don't know - either way he is a Microsoft Regional Director and MVP for Cloud (as well as security), runs courses on cloud deployment on Pluralsight, and has done speeches on Azure and reducing cloud bills, so if a he can get stung it doesn't say a whole lot good about my chances.
The result is that you get a lowest common denominator type of dashboard. And hence a whole industry of providing just a prettier dashboard on top of AWS / GCP / Azure metrics.
Datadog started with a prettier dashboard for Cloudwatch data.
Cloudability started with a prettier dashboard for the Cost and Usage Report.
And also works the other way around. The individual product teams buy development environments to circumvent the console restrictions.
For example, a few years ago, the Redshift team purchased "DataRow".
From TFA it looks like that would be 10 cents per "time series". Or what I translate it to, is 10 cents every 5 minutes (*I think, but I havent used Azure in some time*). $1.20/hour, $28.80/day, almost $900/month. Not too hard to drop that by making the alert less frequent. (edit: I think I saw AU$ there, so maybe it is AU$900.)
Monitoring CPU? Another $0.10 per month. Memory? Another $0.10.
Thankfully, not $900.
As an aside, their (Azure's) pricing docs are written in the same fishy way their technical docs are written (my opinion only)...
They do charge by the MB for ingesting custom metric data, but storage outbound bandwidth is one of the free "standard metrics" built into the platform. So the alert's full cost is $0.10 per month. Since they charge by the "time series," I think that means the OP could configure different alerting thresholds on storage outbound bandwidth, and they'd still only be on the hook for $0.10/month.
The alert emails are way more meaningful (with projected amount in subject for example) unlike generic ones from Azure Alerts, so you see a real alert and prompted to take immediate action.
1: https://cloudalarm.in/Home/Docs/#how-is-budget-alarm-differe...
Also, Azure has an option to alert you beforehand if it looks like you'll go over; struggling to see how your service is any better.
Yes, CloudAlarm, as of now, depends on Azure's pricing data API to do this calculation. It's easy for Azure to do what CA is doing but their threshold based design is a problem, which only prompted me to create this service.
CA also has 'new resource' alarms as well, which are almost instant (a few mins after creation of a new resource), which helps you monitor and fix resource created with unexpected, expensive tiers. This can often happen with automated creation of databases, for example.
I just did it because Azure wasn't doing it, despite people complaining, including me, had faced multiple such issues of unexpected expensive resource got created without intention/knowledge.
In a business setting, you want your service to stay up, at the cost of spike in costs if accidents or mistakes happen.
In a personal project, you want there to be hard limit on cost, and your service to go down if spikes call for it. (I'm relatively sure that no one wants their personal projects to incur a bill of thousands of dollars by accident.)
If you're using a cloud provider I'd highly recommend setting one of those up.
In Azure it's under your Subscription and then Budgets
I truly believe they want you to use a lot of their resources on a consistent, long-term basis; they don't get long-term value from people having short, one-off anomalies, so budgets and monitoring are aligned with their customers - just not total cost of ownership calculations :)
But it's also the case that if they did implement hard limits of some sort, you'd be reading blog posts about how AWS destroyed my project just when it was going big because someone stuck a circuit breaker foot gun in some corner and everything stopped working properly when usage spiked.
I do think there should probably be a hard circuit breaker. It should be simple and therefore inflexible. And it should come with a big warning sign. Still people will get burned because someone will set it, a project grows, and one day it goes off.
While you can raise those limits by request I'm also not sure whether you can actually reduce them again later.
The fact is that there are low end VPS, middle end VPS, high end VPS, and dedicated servers. If you started from a low end VPS, it is very easy to gradually upgrade your VPS.
A $5/month VPS can be used to play for tons of things. I just don't get people who use free tier cloud, unless you just want to learn about the cloud hosting per se.
No you don’t. This is absolutely not a given. Being a “business” doesn’t mean you suddenly have unlimited budget.
The vast majority of businesses are not “web scale” and are better off taking an availability outage than suddenly handling 1,000,000x the normal volume of traffic.
If you are selling you product via your web site and you're suddenly on TV with millions watching and accessing your site, you definitely don't want the server to go downand autoscaling + a bit higher cost would be great.
Making your application scalable is a significant effort that may involve different trade-offs. Your typical Prestashop or Magento e-commerce site will still max out the DB and go down, cloud or not, but with the cloud you'll end up with a huge bill in addition to your downtime.
Engineering your application to be scalable is an option that's often not made for cost/time to market reasons which is fine, but in this case the cloud will give you much less scalability than a lot of people believe.
The post applies to everyone and I’d second it. Ask nicely for a refund in these situations, the worst that can happen is they say no.
Where did they say that “only Troy Hunt shall receive a refund, for only Troy Hunt is a good faith actor, so say we all”?
But Azure’s regular prices are definitely high enough that they’re not a loss leader.
That assumes the only way they make money is on peoples understandable mistakes or lack of care. Doesn't seem to be the case for these services (unlike, say, many gym subscriptions).
It seems far more likely that if they refunded all the well documented issues like this one, their bottom line wouldn't be impacted.
The one thing that is special about Troy is that he is providing a service for the public good but that has nothing to do with being 'good faith' or not.
I've seen several cases on both Azure and AWS that bills got weaved after someone opened support ticket starting with "oops, I just did..."
He did not enable alerts.
Their interests is keeping you as a long term customer. So, they will help you if they can. Unexpectedly high bills like that can end the relation in no time. And 10K is not a lot on a yearly basis. That's a few months of normal usage for lots of companies. So, protecting that revenue is worth something to them. That's also worth realizing when you deal with cloud providers: you are spending non trivial amounts of money on their services and support is part of that deal.
Even if you run a relatively opaque cost structure business like a restaurant, you can still calculate the maximum cost of ingredients for one month, the salaries, energy, etc. if you simply use the "best case scenario" of having every seat at every table booked for all opening hours, with people ordering your most sold dishes. Cloud computing is still leagues above that in terms of cost predictability.
I once worked for small, non-startup software company who pondered moving servers to Azure. The Azure partner shop analysed the needs and came up with a monthly cost "between 30k and 120k per month". They were really surprised the company stuck with their non-cloud setup because "everybody is moving into the cloud!!"
- when we were hit with very high traffic due to a bug or something else, most of the time it would lead to customer outages. Based on the contract some times it requires to pay back because SLAs were not reached. Also an outage could lead to customers canceling the subscription.
We swapped one type of problems with another.
Now if Facebook is down for 15 seconds everyone has heart failure like their life is over.
But the overprovisioned server might still be a lot cheaper than the cloud bill. It can be totally reasonable to have a server running at 1-5% load 98% of the time if you really need the capacity for the remaining 2%.
Also, neither "scaling up" as in "re-deploying the same setup on a beefier instance" nor "scaling out" as in "let's expand to the US and have a server there" is too difficult if the setup is automated (Ansible).
If the restaurant suddenly ordered ten thousand times more ingredients than usual, their supplier would probably call back and say "is that really what you want?" rather than just shrugging and shipping them tonnes of tomatoes with a bill for one billion dollars.
Trying to set a hard number limit ahead of time is hard (estimating how much you'll use, don't want to set a number too low and get cut off plus cloud cost structures can be really hard to get your head around) but that basic level of anomaly detection should be there by default.
Easy way of avoiding this: Don't use shitty hosts that make you pay per GB served and shut you down once you hit your cost limit. Instead get limited by the available bandwidth you have, and clients will just access your server slower rather than being fully denied access.
MS's bandwidth cost a fraction of what they're charging, so it's easy to risk people not paying up.
They assumed it was a mistake and only delivered a single case, which was still 180 limes, but at least it didn't use up our entire food budget.
(Normally I'd expect a phone call or email to confirm, but this was a smaller, local supplier, so they probably didn't have real systems to deal with outliers.)
Troy Hunt didn’t sneak into an Azure DC and install some hardware any more than this hypothetical restaurateur filled a truck at the local fruit market.
Credit card chargebacks, especially.
Though in those cases the billing isn't really complex or opaque, and you _can_ monitor it if you care to check your meter regularly throughout the month. But, for the electrical case anyway, you can't drill into what exactly is consuming watts without either fancy monitoring equipment or potentially tedious investigation.
Just ask Texans.
In contrast, with the cloud the bill is directly proportional to the amount of inbound requests from the Internet, with no out of the box way to implement a limit (I guess you could install Apache/Nginx and enforce a limit there, but doesn't that kinda defeat the whole point of the cloud?).
And (as sibling comment points out) if you’ve got critical household infrastructure that needs gas/electric, you’re not always in control. An unexpected hard freeze could provide you with a big bill next month and you can’t fully control it.
I agree with you that there’s definitely a difference of scale and asymmetry but the concept of unpredictable postpaid bills is not a new thing.
The only thing that would really help were a hard spending limit that stops all services except storage. If your site is important there will be such an amount of user feedback that it is impossible to miss it for a long time.
Or they can fail completely.
And the alerts themselves cost if you want something reliable so you have to weight that against the danger. Pay as you go cloud can be a maze of costing concerns..
> The only thing that would really help were a hard spending limit that stops all services except storage.
Yep. Though that is small comfort if you need to guarantee more than a couple if 9s of uptime, hopefully those with that requirement can soak up the unexpected billing blips.
Sadly, I haven't found a way to do that with AWS
In fact, you could even just change that to any auto-billing service or product and the default for constant-charge services would simply be the amount of the constant charge.
If it's really so 'complicated' that that part can't be done, well, I'm sure it would suddenly become a business priority to cut off use when the limit is hit, if the business can't charge for it.
Some things are legitimately very difficult to meter in a real-time way with adequate performance. Imagine if the S3 server had to do a DB query to lookup your account balance & limit on every HTTP request. That'll completely kill performance and availability (what if the billing DB is down? Should that DB now be replicated across all regions? etc).
The reason a lot of cloud alerts and metrics lag is because billing is done asynchronously by parsing logs way after the actual usage occurred. Of course, the real solution here is to just not bill for things you can't easily measure & limit especially when those things are super expensive, but cloud providers' C-suites have to eat & pay their bills somehow.
Actions weren't there last time I checked (few years ago).
Any enterprise will not want any limits because of spends, they would be lot more pissed if service was pulled because spending cap set by someone sometime in the past is now exceeded. Likely is why such feature is optional not mandatory.
Excess/unexpected billing would be negotiated in typical sales cycle discussions. Making a default hard cap however would result in a lot of senior people are going midnight calls for emergency budget approvals, management would get annoyed by that.
Admittedly I don't have experience working with very many devops /SRE teams, however I have never seen any enterprise vendor relationships were spend is hard capped for b2b sales with centrally managed procurement.
A lot of people have provided tons of examples of follow ups by vendors in this thread where novel large orders are double checked. AWS could provide any manner of preventative policies and cost controls but they choose not to.
Given the choice between blowing 2x past the planned opex and an outage, yeah, there are plenty of applications where the latter is preferable.
The dumb solution for that is to exclude persistent storage from the limit.
The nice solution for that is supporting both "runrate" and "consumption" limits.
Using a runrate limit, spinning up an instance, creating a file, etc. allocates budgets for running it continuously, which is released when shutting it down/deleting it. Hitting the limit prevents new resources from being allocated, but keeps existing ones alive. This should be used for persistent storage and instances used to handle base load.
Using a consumption limit, the resource is shut down when the limit is hit. If the shut-off is delayed, the cloud service eats the overage, since they control the delay. This should be used for bandwidth, paid api-calls, and auto-scaling instances.
The user should be able to create multiple limits of each kind, and assign different services to such limits. Alerts when going near the limit can help the user raise it, if that's their intention.
For consumption, it might also make sense to have rate limiters, which throttle after a burst budget is exceeded, similar to how compute works on T instances on AWS. But those probably only make sense for individual services, not globally (e.g. throttle an instance to 100 Mbit/s after it exhausted its 5 TB/day bandwidth allocation, or throttle an API to x calls/s).
AWS has decent tools in this regard, but it pales compared to Oracle. Azure is a product I've never used with any scale (just small projects), but the fact that it actually costs money to setup alerts is gross (and morally reprehensible). Even if it's a trivial amount, that alone just sours the product in my eyes. I mean, already Azure is pretty uncompetitive unless you're running on free credits, as Troy apparently is (purportedly some $13K per year, so unsure what the pitch for donations to cover a bill is about).
Oracle Cloud has an enticing free tier, but I'm too afraid to use it because it requires a credit card and I don't see any way to put a monthly cap on my budget. (I'm sure hobby projects with ~$5 - 10/month budgets isn't their target market, but I can dream :)
Edit to add the page I was reading: https://docs.oracle.com/en/cloud/get-started/subscriptions-c...
I think that the cloud provider business model that allows for uncapped maximum costs is a bit of a commercial dark pattern. What makes it somewhat more nefarious is that it is relatively easy to blame the customer.
I’m not surprised that the cloud providers are quick to refund users as it’s likely that they only do it in a fraction of cases and it buys a lot of goodwill.
It would be interesting to try and design a cloud that supports OutOfMoneyException’s with gradual degradation and capped liability for costs built in.
I don't actually believe so. Cloud providers are known to refund bills incurred by mistake. They make so much margins on legitimate usage by big companies & startups that it's just not worth burning developer goodwill & potentially waste efforts trying to collect a bill the customer legitimately can't pay (and will guarantee he will never use nor advocate for your service again).
https://azure.microsoft.com/en-au/pricing/details/bandwidth/
My conclusion Troy still doesn't know how much he is paying.
Actually, wow it seems AWS is also the same price as Linode and DO for egress. While Linodes and DO do come with decent free bandwidth, this is a surprise to me.
AWS charges $0.09/GB, and Azure charges $0.0875/GB.
Maybe Troy Hunt gets a discount for being a Microsoft Regional Director and MVP. (Neither of which make him an employee of Microsoft, confusingly enough.)
https://docs.digitalocean.com/products/billing/bandwidth/
https://www.linode.com/docs/guides/network-transfer/
https://aws.amazon.com/ec2/pricing/on-demand/
https://azure.microsoft.com/en-us/pricing/details/bandwidth/
https://azure.microsoft.com/en-au/pricing/details/bandwidth/...
The AUD $0.014/GB is only for data transfer between Availability Zones.
When everything was moved to production, URL went live, nobody ever did any kind of bandwidth checking, caching, no CDN, no cost tracking. $10,000 in our first week. That's about 1/4 what our total spend on the co-located servers was for the whole year. Boss flipped his lid and wanted to kill the new guy who was on the project.
After about 2 years we got rid of all the co-located stuff and were spending about 1.5x, but we had more apps, they served heavier pages, etc.
We overspent quite heavily on our on-prem stuff for a game I helped launch, for political reasons the next game ended up running on the cloud.
The price was roughly 10x before discounts. With our heavy discounts and a wide amount of slimming down/cost optimisation (easily 3 months of work) we got it to 2.3x
There will always be a need for sysadmins/cloudops/devops for that environment, so we didn't save any headcount either.
I can't imagine getting anywhere close to parity in costs, Functions-as-a-service ended up costing more than compute instances too so we went back to compute instances in places where we thought we'd get away from it.
That said, it was a lot nicer to use!
It is important to remember that not all cloud providers participate in it. For example, in Hetzner Cloud, they explicitly provide the maximum amount you are going to pay for a given instance or service in a given month. You are guaranteed not to pay more. Everybody knows why Amazon etc. refuses to do it this way.
"Your account has exceed $100 spend. Reply 'SHUTDOWN' to shutdown all services, 'STOP ALERTS' to never see this alert again, or 'DOUBLE TRIGGER' to double the alert trigger value to $200."
$100 is arbitrary, it could be any nominal sum. The idea being that the user can double the alert each time they get it just from SMS. I bet 95% of users would double their alert limit to a comfortable point. The other ~5% will be power users who customize their alerts.
The idea that these companies couldn't know what limits customers want is kinda silly. We can use the same techniques for alerts that we use in algorithms for expanding vector storage, for example. We can "amortize" alerts, so to speak.
We have some recent case studies where we've successfully reduced cloud costs by 95%
https://www.cloudexpat.com/case-studies/
hi(at)cloudexpat.com - happy to help!
Or someone maliciously bypasses CF cache e.g. by parameters.
Cloud just is not suitable for any kind of volume egress. It's a death trap. Like going on vacation with data roaming enabled.
> I removed the direct download links from the HIBP website and just left the torrents which had plenty of seeds so it was still easy to get the data. Since then, Cloudflare upped that 15GB limit and I've restored the links for folks that aren't in a position to pull down a torrent. Crisis over.
But I feel like Dr Strangelove here. Of course, the whole point of a torrent on a cloud service is lost if you also provide a raw download link.
Also providing a download link is tempting, but can easily cost (for a 17GB file and growing) up to US $3 per click.
Even off of their premium global network it's over $2 per click. The cheapest in Microsofts entire egress table would be $0.68 per click. (but that only kicks in after you've spent way more than $9400 in cheaper tiers in a given month)
Egress kills you, in cloud. "Oh, cloudflare probably caches most of this" is not something I'd recommend.
- Patching - Remediation, Monitoring, day0 response
- Security Information and Event Management - exports, alerts, OS configuration
- OS/Application Hardening - Encryption, Password/keys rotation, CIS/other baselines, Drift Management
- Backup - Encryption, (don't forget your passwords/keys are changing), retention, data protection compliance, monitoring, alerting, test days
- High Availability - replication, synchronisation, monitoring, alerts, test days
This is just the tip of the ice berg, if you operate in an environment where Insurance, Reputation, Regulatory Compliance, etc.. are important, then it's easy to see why PAAS solutions are desirable.
Mice cried and stung themselves, but kept eating the cactus.
Most CDN providers have a lot of machines out on the edges of their networks, and it's understandable that they don't stuff these machines with large disks, likely preferring smaller faster SSDs. But this is a very common pitfall of CDNs that needs more attention, along with messaging on the dashboards and settings pages.
I've had problems with no warning on Cloudfront, Cloudflare, Bunny.net all from not realising that my files were beyond the CDN's cache size limit, but none of them seem to do a good job at surfacing this other than "talk to customer support".
Cloudfront does list the max size clearly in the limits and quotas page, though, and if you front your S3 bucket with Cloudfront, you could turn caching off and still get the discounted bandwidth out rates (S3 -> Cloudfront is always free, even if the file is fetched every time).
I see S3 is initial $0.09/GB, going down to $0.07 after 50TB or $0.05 after 150TB.
Cloudfront North America is $0.085 for first 10TB; but $0.110 and up for other regions. going down to $0.060 north america after 100TB, and okay $0.025 after 1PB. (but $0.050 and up in other regions even after 1PB).
So okay, Cloudfront gets cheaper egress at large scale, I guess. By about 50% though, not an order of magnitude, and could be much less depending on region.
A large bill is probably chump change for someone like Troy, for others it's a year or two of savings. The risk is not worth it.
At $20 bare metal is not easily possible, the lowest prices I have seen are usually 40-50 and above. Howveve you can get a VPS with unmetered bandwidth and no other costs at your price range [1]. The price is still fixed some performance variances may be there, at $20 minor variances are unavoidable.
AUD $0.014 is roughly USD $0.01. Which I thought was reasonable. But on [1] only "Data transfer between Availability Zones(Egress and Ingress)" cost $0.01. Do transferring from Azure to CF count as that? Other Internet egress (routed via Routing preference transit ISP network) starts at $0.08
I hope someone from Azure CS could give him a custom discount.
It is also worth thinking, the cost HIBP saved on Cloud / Serverless over the years could have wiped out ( if not more ) by this single incident.
[1] https://azure.microsoft.com/en-au/pricing/details/bandwidth/...
To be clear - we would not have been able to catch this one right now :'(
Would love to hear thoughts / brainstorm ideas - is there any way we can proactively catch these types of cost spikes?
Setting up limits and alerts as part of the system creation is usually the best strategy.
Cloudflare has the same model, but they distribute the costs. The vast majority of people never use anywhere close to their share, so they subsidize the outliers and the free tier.
https://blog.cloudflare.com/the-relative-cost-of-bandwidth-a...
Convenience always costs money, there is no (big) cloud provider doing it out of their own pocket or rather not optimizing for huge profits.
It's the same as with any other service, really. So I don't understand, why some people assume it would be different here.
(Note: I am not saying that Troy Hunt assumed this, but I know people who go to the cloud because "It's cheaper". It was never cheaper, on no project I worked on. It was more convenient, but in the end it was more expensive mostly)
No matter what the traffic is, The first thing to do with any cloud service provider is to set the budget alerts according to our wallet, be it one with credits or otherwise. At this point, I don't even try any new cloud service provider who doesn't offer credible budget alerts.
Another key takeaway is,
> Huh, no "CacheControl" value. But there wasn't one on any of the previous zip files either and the Cloudflare page rule above should be overriding anything here by virtue of the edge cache TTL setting anyway.
Even this could blow up. All cloud service providers set the "CacheControl" to "No" and if we would want to cache something which is not cached by CF by default e.g. *html using Page Rules then we need to set CacheControl (e.g. max-age) at the cloud service provider end too.
P.S. I've written about these recently on my blog titled 'Saving Cloud Costs'[1] from a frugal solopreneur PoV.
It's my opinion that it's better to work with known limitations and optimize for them.
In the case of bandwidth, work with a fixed pipe size, or do the math and set up a QoS that implements a throttle to avoid exceeding your bandwidth allotment.
This happened to Murfie a couple of years ago, and that's why I had to step in to try to fix things. I'm still trying, and there are still challenges, but I won't allow landlords and cloud costs to disrupt things again.
It's time for cf to work a bit on its UX
Uh no - it's on cloudflare and azure. Why don't they have a global setting that says Max Charges Per Month: $X and it just shuts down when it hits that number? This is why I don't really like using big cloud services like this.
Turns out I wasn't setting x-ms-cache-control when writing all the blobs, so that's a win right there.
(interestingly, it appears that rclone, which I was in the process of moving to, doesn't do that, so I might have to keep my custom Azure storage library around)
$ shard="$(echo "${sha1}" | cut -c 1)"
$ cdb -q pwned-passwords-v8-sha1-${shard}.cdb "${sha1}"
He should have known better that there is a risk, that you don't know some detail that costs you a lot of money.
Cloud Bandwidth is soooooooooo expensive. If there is a risk that you have to pay this, please us a provider like Hetzner with fixed costs. If you like your serverless things, just host the big files at Hetzner.
It's suspicious that cloud providers STILL don't have any sort of "circuit breaker" infrastructure for this sort of thing - yes, you can set up alerts, but you can't say, "shut the whole thing down before the costs go above a certain threshold".
Kind of sad that service we are accustomed to using, various software integrates it (whether using HIBP API or downloaded pwned passwords archive) - is on a shoulder of single guy that now has to pay for his mistake.
Great that Cloudflare helps him with the service, otherwise who knows if we had access to HIBP at this scale?
Fact is that stuff like this can happen. Consider how many variables are in play to determine the final cost of a cloud service it is very much a double-edged sword. Sometimes you cut yourself unintentionally.
So now we all learn from this, I suggest we help him out.
If you're not Troy Hunt or another celebrity with special access to Cloudflare -- I don't think you really have access to Cloudflare to do a lot of work with you to ensure that your data gets cached and your egress is minimal, for large files on a very cheap cloudflare plan. (Based on the costs reported by Hunt as catastrophic, I don't think he's paying cloudflare for a large enterprise plan)
(Also, it's unclear if caching large data like this is even within the ToS of Cloudflare?)
I don't think Cloudflare promises to cache any particular URLs for any particular amounts of time (except no greater than cache headers etc; but they don't promise never to evict from cache sooner; they evict LRU according to their own policies). Cloudflare's marketed purposes include globally distributed performance, and security. I don't think they include "saving egress charges by long-term caching your data".
I have a much smaller project, but egress charges for data are an increasingly large part of my budget. I've been trying to figure out what if anything can be done about it. I wish I had a guaranteed way to get ultra-long-cache promise-to-be-within-ToS for very large data files from Cloudflare for a affordable fixed-rate price. (Maybe I do? But just haven't reassured myself of it yet?)
> In desperation, I reached out to a friend at Cloudflare… I recalled a discussion years earlier where Cloudflare had upped the cacheable size… Since then, Cloudflare upped that 15GB limit…
Since I'm looking for solutions for this same problem (delivering lots of data at very cheap prices), I am finding myself a bit annoyed that Hunt is talking about how he solved it, using tools/price-levels not available to most of us who don't have his level of access due to position.
Interestingly, MSN/Azure is part of the "Bandwidth Alliance" with cloudflare, which initially one thinks means there are no egress charges when delivering to cloudflare. (That is what it means for some other alliance members like backblaze). But that's clearly not the case or this story wouldn't happen, right? Turns out Azure gives you a fairly small egress discount when delivering to cloudflare, and only if you set things up in a non-standard way.
as long as your human admin costs are lower then cloud services
I get that everyone has an obsession with dirt cheap providers instead of cloud solutions like aws/azure. But that doesn’t mean it’s better. Everything has pros and cons.
What would happen if a credit card limit was exceeded, a site would just stop working?
But still, couldn't help to get the following lasting impression after reading it: these days being able to click around the UIs of the cloud providers should be a billable skill by itself.
I took a good course on pluralsight about AWS and the first lesson was to setup a billing alert.
What will hard limits will do to your infra? You can't take down / suspend DBs, EC2s, etc... Just because you set a 1k USD limit and that's it.
Alerts are the 1st thing you should setup IMHO
You (the cloud provider) can shut down VMs, block access to all services, and just retain the content in storage until the bill is resolved or the account is permanently closed. The cost would be trivial as storage is dirt cheap.
AFAIK Heroku shuts down your stuff if your Dynos are overspending :P
I really appreciate the work that Troy is doing, but seeing much needed money ending up and Microsoft or Amazon leaves a bitter taste. I hope at some point it will become cool again to just rent a VM or dedicated server for small projects and stop throwing so much money at the already richest people in the world.
EDIT: Apparently it was hosted out of US West, so I agree that the actual data cost would probably be a lot less.
As far as I am concerned, I just don't understand why people use cloud services.
Edit: Consider this article, and Geoff's statement about Azure credits.
https://www.theregister.com/2021/04/21/microsoft_revokes_mvp...
How is using cloudflare okay in this then? Cloudflare is also not Azure
(Edit: fix spelling)
Well, it might also come with contacts in the billing department.
> I'm going to miss the $13,000 USD (yes) a year in free azure credits. Just remember this amount of money when you are reading content about "how good azure is" and "what the latest and greatest is" from influencers and community leaders here on social media...
Next thing I see them being slapped with $700K bill and managers running like headless chickens all over development floor and yelling to turn off every VM, hard drive, database / whatever either resources.
From the users point of view, Cloudflare will frequently stop you from accessing things and introduces more single points of failure in the internet infrastructure. But on the good side, they have pretty good edge endpoints so your browsing might be a bit faster, when they allow you to browse.
They also offer Azure CDN, as a competing product. But I don't know if anybody takes it serious or not
= Azure has an integration to use cloudflare for the cdn.
I'd be suprised if his Microsoft Regional Director and MVP status isn't worth much more than 4 figures to him.
Those seeking to initiate engagements with Troy might care more about the fact that he pops up on HN and other high profile tech outlets frequently and the visibility of Have I Been Pwned, but the Regional Director status probably helps a lot with getting some of these engagements signed off.
He probably also receives significant subsidies from Microsoft as well.
"Firstly, I always knew bandwidth on Azure was expensive and I should have been monitoring it better, particularly on the storage account serving the most data."
...and he didn't have simple monitors in place to alert him of uncommon billing spikes.
I get your point, if he's not OK with using Hetzner how is Cloudflare any better? It's not. But the reality is Cloud operations are a fine dance of weaving services together to realize all of the heavily advertised savings. I'd argue that a lot of Troy's projects that use all of the cloud native functions could have also been implemented on much more standard stacks and, likely, been just as cost and performance effective. But that's not going to get him the advertising for Microsoft.
You want to waste money? Hire a car, with a driver, when you need it.
Want to save money. Learn to drive.
You always pay more for outsourcing stuff, a lot more, than doing it yourself.
You can buy 1000x the processing power, by buying baremetal. You can get 100,000x more bandwith for cost, when not using the cloud.
People think baremetal is hard. It isn't. It does take knowledge.
Hiring a personal car is more expensive because you are hiring a personal employee.
That said, I still argue for personal autonomy alone learning to do the thing is better in general, but I don't think it's because it's cheaper in all scenarios. And to your point some or maybe even most cloud services are more expensive relative to their self hosted versions.
Let's use your driving example (because car examples are always great!)...
>You want to waste money? Hire a car, with a driver, when you need it.
>Want to save money. Learn to drive.
This is true. You can save more money if you need to drive often if you own a car. But there are two scenarios that it still makes sense to rent.
1) What if you need a car in a different city? You just flew from JFK to SFO. You already have a car in NYC, but need one in SF. You're not going to buy a car in SF that you'll need to sell in a week. Sure, if you're going to be there longer, you might consider it, but then you're still carrying the costs of two cars.
2) Sometimes you need a truck. Maybe you have an IKEA run to make to get a bunch of desks, or stop at the hardware store for a few dozen bags of mulch, or ... But sometimes you just need a truck to get the job done. You could just buy a truck and be done with it. But trucks can be more expensive than a compact car, and they definitely have higher fuel costs. In this case, you'd probably be better off with a fuel efficient (or electric) compact car and rent a truck only when you need it.
This is how you save money with the cloud. But you definitely don't save money when you effectively rent a truck to drive to work everyday (even if you are in construction). There is a cost to renting -- it is more expensive on a per-use basis than it is if you buy. Cloud servers are more expensive than bare metal -- if you're constantly using them. It is only cheaper when you stop paying for the parts you don't need. And that also takes expertise.
Once, at a new job, I inherited a cloud server. It was costing us a ton of money per month and running 24/7 because the person who set it up never turned it off. After 3 months of those costs, they could have bought a new server with no other renting. They paid for a cloud server for three reasons: 1) they had no experience with hardware, 2) it was a pain to setup local hosting, and 3) it was faster to get running without waiting for a vendor to build a server, deliver it to the datacenter, etc... These were real impediments to the first person and the cloud server helped to get them moving. They just didn't have the longer term view of what their decision was going to cost in the long term.
The first thing I did was order a new server and make friends with our datacenter ops people. And now the only thing we really use the cloud for is archival (write-once, read-never) storage. If we ever really need these data, it will be super expensive. But, if that ends up happening, we'd be happy to pay the cloud tax.
Cloud was made for people who don't have the time, talent or desire to build and manage it in-house. You pay a premium for that convenience and that premium scales with your business growth via IT resource needs. I think that's what he was getting at in his analogy.
Isn't that exactly how companies use the cloud? Sure, there are contrived examples where the cloud is cheaper than self hosting. But the common case is that companies "use the cloud" by putting 100% of their infrastructure and hosted products in the cloud. That's what is meant when you say "X uses the cloud".
a - replication (within region / AZ at least) b - 0 software to maintain (no need to frantically patch apache / SSL / whatever) c - super quick set up / management / logs / etc
So, yes, bare metal is (on a cpu cycle to cpu cycle / GB RAM/HDD/Bandwidth) level cheaper, but TCO can be waaaaaayyy higher.
You have to do a lot of things right to get that Cloud Value, as the author of this blog post has shown. You have to do a lot of things right to get value out of on-prem bare metal as well, but those things are generally well-known, standardized, have less moving parts, and people with decades of experience and knowledge of best practices. The opposite of the current cloud landscape.
TCO is not a straight line.
This.
I always wonder how much of the "clouds" success (economic, that is) would have materialized, if the marketing term never got traction, and everyone just called it what it really is: "renting someone elses hardware without physical access, and less, if any, control over how the stack works from the metal up".
In the good 'ol days, when people wanted to put a service online, they rented the racks at a colo, and either stuffed their own hardware in or, worst case, used rented hardware.
Did that require some basic familiarity with hardware? Yes it did. Did people need to know how to setup, configure and administrate a LAMP stack? Sure. Was it guarded against sudden loadspikes by god-knows-how-many layers of abstraction? Nope.
But it worked, and surprise, in 99% of cases, it was perfectly fine if a website ran at sub-optimal speed for a few hours, or went down every now and then.
And the dirty little secret is: It still does, and it still is.
I delved into this pretty thoroughly last month - https://medium.com/@rykrk/everything-is-just-build-vs-buy-d7...
So is a LAMP stack on a dedicated machine.
> Plus, the more of the services you use,
The thing is, most webapps don't use a lot of services. Backend-Logic in whateverlanguage, a database, and a webserver. Maybe hooked up to some CRM system. That's it for 99/100 webservices.
Yes, the services cloud providers offer are amazing, they are complex, and it is natural for developers to be fascinated by complex things (I know it is for me). But it's important to realize when simple is simply enough.
Honestly, the question you need to ask in regards to cloud is a relatively simple one: Can I hire a sysadmin for cheaper than using the cloud?
The answer to that, once you start using enough resources, is more often than not yes.
Sure, it takes a while to get to that point, but eventually you will reach break even and it would be cheaper to do it yourself/have your employee do it.
I have been on both sides. Large media production companies with very large amounts of fast and redundant storage located on-prem. These range from local attached RAIDs to large shared SAN pools. Their clients also tend to be the types that sue the crap out of you if any of their content is seen by people outside their control. Switching to cloud solutions was (still is) a huge uphill battle. However, the cloud storage needs are no where near the same (not editing content from s3), but storing approved masters for distribution totally makes sense for cloud. Now that the content is in the cloud, why not perform actions on that content in the cloud. Faster deployment, better equipment, blah blah. Next thing you know your entire workflow past editorial is in the cloud. You start to analyze your expenses and compare them to on-prem amortized expenses and you see that it could be cheaper on-prem. Also, take into consideration how long it takes to bring up that new data center. You also have to look at bandwidth expenses. Bandwidth to a new site not directly on the backbone tends to be expensive for non-residential connections. The additional power expenses of that new equipment plus the cooling is also a new expense. Power redundancy you ask? $$$ Now, you need that sysadmin and possibly a small team. At that point, you go back to your cloud rep, and renegotiate fees. You have now created an entirely new department at your company on managing the on-prem.
> Want to save money. Learn to drive.
Oh please. As if learning to drive is the end of expenses. If you finance a car, you have monthly payments. If you don't, then you have periodic recurring maintenance bills. You always have fuel charges. You always have insurance charges. You periodically have parking charges.
I know how to drive, but do not own a car. From time to time, I hire a car, but it no where gets close to costing me the amount of owning a car would.
but i think where this analogy breaks down is that if i run a service, no matter how many users, peaky or not, at least 1 server always needs to be on, not "from time to time".
... I run a junker. That is to say a car that will go the dump as soon as it requires any significant expenditure, and the combination saving of not having to finance it, and most years minimal or no repairs, and only needing third party insurance makes it significantly cheaper than renting.
in practice I spend maybe 1k every year for cars. primarily for vacation. which owning a car wouldn't absolve me from spending.
In my bare-metal-hosted projects I can afford to do a lot of things that would be a major no-no in the cloud because I have so much hardware resources I can save development time doing things inefficiently and still come out ahead in terms of costs.
I haven't checked, but are the prices for Azure CDN relatively competitive with Cloudflare? I think you'd probably get similar savings going that route, and it would all be Azure.
Seriously, yeah, if he's an MVP, he'll be fine.
Getting metrics on that is not a hard problem, there are various projects that are relatively simple to set up.
If you want to make it easier manage resources, metrics out of the box, and avoid (hoster) lock-in then I'd use a hyper visor distro like Proxmox VE (disclaimer, am a dev there) or the like, and you can migrate (or backup/restore) VMs or Containers easily to other providers. That gives you a (relatively) simple web-interface to manage most things and also opens the possibility to just add a second or third dedicated host down the line to scale out, if those new hosts are in the same DC or have a good interconnect (latency wise) you could even cluster the nodes.
I can set up Proxmox VE as hyper visor, some container for each DBs, load balancer in front and some app in about an hour max from scratch, with good testing and some bells and whistles, and here I really do not want to brag or the like, as such operations are not my job to do at all, I only know because I do that occasionally for some tests and for some private infra I just maintain out of interest - so I really want to say, if some operation-dork can do that, the engineer you hired should be able to do it at least as quick.
But yes you're right in the general point, upfront setup and frequent maintenance is naturally something you need to price in. I just think that if you have that many different parts with complex coupling to induce such a huge maintenance effort required to keep your product running, the cloud offer may not really be your salvation and just delay the fall while costing all the more.
Everyone does something for the first time once. Just because someone has not set up a hypervisor before doesn't mean they're inexperienced.
> I can set up Proxmox VE as hyper visor, some container for each DBs, load balancer in front and some app in about an hour max from scratch,
And I can spin up containers + load balancer on AWS in less than five minutes. That doesn't mean that it's just an easy thing to do. (although, this specific example is).
> upfront setup and frequent maintenance is naturally something you need to price in. <...> the cloud offer may not really be your salvation and just delay the fall while costing all the more.
Agreed 100% on both counts.
Wait, what? If you never did something, then you're unexperienced, ain't you?
Experienced != Knows 100% of things.
I'm sorry, but all that stuff you describe doesn't bring any business value. My customers don't care what hypervisor I'm running, so I don't care either. PaaS means someone else deals with it, forever. The last time I had to employ an ops (or devops) person was 2007.
Also, it doesn't take a week.
You're assuming that this is for recurringly set up infrastructure. Sometimes infra is set up once and maintained, othertimes it's set up and spun down. It's also not always automated. The time spent automating something like that might not be worth it in the medium to even long term.
> Also, it doesn't take a week.
The actual amount of time it takes doesn't matter; if it's a day or a month. what matters is costing the time spent on setting it up and maintaining it, and pricing that against AWS costs.
For a large company, it's not about $ cost, it's about risk management and avoiding cost centers.
I'd love to hear your reasoning why people who aren't European would prefer to avoid European service providers.
I'm generally a Hetzner fan as well for global services, but I can see the point in avoiding Hetzner (for example) if all of your users are in the US, since Hetzner only offers dedicated servers located in Europe (Germany and Finland if I'm not mistaken). Generally you want users to hit servers that are close to them, so something like Vultr would be better if the scenario mentioned before applies.
They also have dc's in the US
I didn't realize it was limited to their cloud offering. Nice one!
I'm sure there becomes a point where cost of (hardware + maintenance + staffing) > (cloud + staffing), in which case sure crack on. But like you, I'll stick to a rented server for my stuff.
I'm a major cloud skeptic, but there's a certain class of giant enterprisey companies that are never going to be able to attract good IT talent, and if they "just throw money" at the hiring problem they'll be innundated with slick imposters.
I think cloudy stuff lets those companies outsource a large chunk of something they'll never be good at. The cavalcade of Microsoft/Cisco certifications were an earlier decade's attempt at solving the same problem.
Even larger companies can work well with that model, traffic also tends to be cheap enough that you can spread across different vendors to avoid lock-in. And in that case, your sysadmins can sit wherever they want, no need to be physically close to the servers.
Also, as there's much less knowledge to be a dedicated server provider, competition is strong and prices are comparably low.
Also it is a major pain point getting anything done with IT operations.
Like the Oracle database server that half the department relies on stops responding on a Friday morning and it takes all day to determine the hard disk is full and fix it. I had never before worked at a company where this happened multiple times.
Or operations saying they were unable to restore a windows server hosting a database server and now everyone has to scramble to update their connection strings because operations somehow cannot use the same domain name for the new machine.
If you're Netflix, cloud is probably not that much more expensive than owning your own servers. Maybe even cheaper. But you're not getting Netflix prices.
But even if you're small fry you should however start regularly talking to your provider and go through a regular cost-cutting exercise and talk to them about how you're looking at provider Z and have been asked to cost out managed servers and on prem options.... You won't need to get very big before that starts paying off.
If your competition is doing this and you're not, and hosting costs starts becoming a big part of your cost base, you won't be able to compete.
Long term I think we're going to see disruption here to the point of startups failing because of competitors copying their idea but being better at driving down hosting costs by not being afraid of going to dedicated hosting or hybrid solutions (hybrids are my favourite - if your stack can be deployed semi-transparently both on dedicated servers and cloud you can go much closer to the wire on your dedicated servers by being prepared to spin up cloud instances to take care of spikes; ironically having the ability to spin up cloud instances makes relying on cloud services even less cost-effective)
I'd also expect to see more "hybrid" cloud offerings with companies offering you operations-as-a-service by giving you a virtual cloud type interface where they don't actually own a cloud service themselves but helps you abstract away cheaper hosting providers. You can already find plenty of people who'll e.g. run Kubernetes setups for you, so taking the step to do more cost-optimization on the backend is natural (and I'm sure there are people who'd do this for you today - if I was still doing contracting I certainly would be offering that - and maybe someone is already wrapping it up as a service offering; I haven't kept up on that market)
I used to spin up dedicated servers and then put an overlay network + a simple set of tools to spin up containers on them years before Kubernetes etc. was a thing, and we'd have a "global" (we had VMs in Asia, dedicated servers in Germany and colocated own servers in the UK) unified deployment mechanism that let me spin up containers wherever with a one-liner. Having a few extra dedicated servers with spare capacity standing by still made the whole system far cheaper than e.g. AWS, even if you attributed my entire salary towards it (I spent nowhere near all my time keeping that running).
It's easy enough to find consultants that can set up systems like this that abstracts away the dedicated hosting providers so you can mix and match and move with ease - especially today with options like Kubernetes.
If I was to go back to doing consulting I'd probably look at finding a way of packaging this kind of offering up behind lots of marketing speak and offer some sort of "abstract" hybrid private cloud layer on top of a choice of dedicated hosting providers to make that kind of hosting palatable to execs who refuse to believe the cost saving potential because they've never dived into the actual numbers (oh, the amount of time I've spent building out spreadsheets with precise cost models that'd get promptly ignored because someones had heard from a friend that company X swore vendor Y was cheap and believed it blindly)
Companies that have made a name for themselves by outsourcing to the cheapest IT contractor that will promise them the moon and fill the seats with barely warm bodies? I was one of those bodies so I know exactly why they can't attract talent - they don't bother, and don't reward it. They treat IT as a cost center and are surprised when they get disrupted. The only good options in those companies are to work on the business side or worst case as a project/product/program manager interfacing with the warm fungible contractor bodies.
Many Enterprises are only alive because of inertia and goodwill from earlier decades.
It’s just a shame they were permanently out of database servers with SSD storage, and for some reason couldn’t provision more for over a year.
There are smaller cloud providers, rented VMs, rented dedicated servers and rented colocation space.
Mid-sized companies can get cracking deals (like 10% cost) on major cloud providers.
The biggest issue, though, is how few people are aware they can negotiate with their cloud provider. I've seen so many places just pay the sticker price without even trying to get discounts.
(Conversely, I once got a contract to do zero-downtime migrations first from AWS to Google Cloud and then to Hetzner so a startup could launch on AWS and spend the huge amount of free credits they'd been given there, then migrate to Google Cloud to do the same, and then finally move to Hetzner once they had to actually start paying; relative to what they'd have to start paying if they'd stayed on either AWS or Google after their credits ran out the cost of having me do the extra setup to handle that was covered with ~2-3 months of their savings)
I kept being asked to price out a migration to AWS, and we kept coming up with 2x-3x the cost. Part of the reason was that we could pick and choose servers that fit our workload in a way we couldn't with AWS, and partly the absolutely insane bandwidth prices AWS offered.
I use AWS. I like AWS for the convenience. But it's a luxury that is ok when you're either small or really high margin, and you're paying massively over the odds for that luxury.
The reason these services get away with being so expensive is that people massively overestimate the complexity and don't bother actually getting quotes from people or companies to manage these services for them. When I was doing consulting my biggest challenge in offering up alternatives to AWS was that people were so convinced AWS was cheap that even when presenting them with hard data they often didn't believe it. For me it was a mixed bag - I tended to make more money off the clients who stayed on AWS as they usually needed more help to keep an AWS setup running than those I migrated to managed hosting setups, despite paying more for the hosting too.
I’m a huge Hetzner fan, and their cloud offering is definitely growing but still isn’t as convenient and featureful as it could be (and they don’t share their roadmap currently so hard to tell what they’re working on next).
I’m trying to do something about it though, working on Nimbus Web Services[0]. In my mind all we need is something to bridge the managed services gap and make it very easy to set up the basic 3 tier app with some amount of scale/performance elasticity!
[0]: https://nimbusws.com
apt install nginxHonestly, they're not even connected/brain-dead-easy for a vendor like AWS particularly -- you still have to click around a lot or write a bit of terraform/aws-cdk/etc when all you really want to do is throw a folder or zip file over the wall and point it at a domain.
There are tools like Ness[0] out there which look like a breath of fresh air but there needs to be more tools like that where the supported backends include a cloud like Hetzner/Leaseweb/OVH.
I'm in no way a sysadmin and have set up these configurations manually in less than an hour for side projects. Cloudflare tunnel also allows you to lock down the server for everything but ssh with pubkey auth so the attack surface is really small.
Actually hosting files is super easy (Caddy is awesome, NGINX is awesome), but it's even better when you don't have to set up the server at all, for example just turn on "HTTP access" on a object storage bucket for example. So this is another place Hetzner kind of falls short though they do have hosting options[0], so basically the ideal solution here would be to deploy a simple Hetzner app (caddy/nginx or the hosted options hetzner has), set up a cheap CDN (Bunny, Cloudflare, etc) in front of it, and save money that way. If the bill is still too high just take the penalty or bias towards one geo (germany/US).
I was less talking about the difficulty of getting a server up and more about the CDN bit of the issue to make loads blazing fast!
[0]: https://www.hetzner.com/webhosting what you want is latency reduction. Usually what sites like Vercel and others give you is way faster loading time by putting stu
I agree! Cloudflare probably won't be this cheap forever but like I said I think that's the optimal solution, with the option to cut over and take the latency penalty if costs are out of control.
Latency isn't particularly relevant for this, and it probably isn't relevant for most hobby projects.
So that leads me to my question for HN. Have we completely abandoned non-HTTPS, particularly perhaps for the use-case of server-side caching of HTTP content? Also, isn't this a valid use-case to not use HTTPS and to re-enable that sort of functionality at the network/ISP level?
For companies the benefits are the abiltiy to get new servers at a click of a button and get rid of a server. For example, asking the ops team to setup a snapshot of a database for a few hours while I do something is super useful.
There is also the ability to use autoscale and other stuff to automagically scale your system to handle traffic peaks. With dedcicated servers you need to always have those resources available. It's attractive to managers that they're only paying for resources when they're using it.
There are also managed services like DynamoDb, Lambda, S3, etc that can make things easier and reduce your sysadmin work. And allow you to get up and running very quickly.
Obivously, a major downside is that the pricing is extremely vulnerable to spikes like this. I think we see an article like this every 3 months or so. This one is rather tame compared to some others that were 10x as much for a 24-hour period.
I can snapshot a database disk with a click of a button and restore the snapshot with yet another few clicks.
I have 1.5 TB of highly available disk space, 40 cores of full CPU power, 160 GB of RAM, & dynamically provisioned IPs for metallb. For only $130USD a month. For the same price in Azure, I had 6 CPU cores & 8 GB RAM.
But let's say need 4x vCPU: 72, Memory (GiB): 144 for 4 hours. Or you need that 12 hours a day but for the rest of the time you need 2 cpu and 4 GiB of memory.
You need to handle traffic spikes such as TV traffic.
Yes you could self-host a cloud env but you can't scale your resources the way you can with cloud.
And that is literally the largest AWS Elasticsearch cluster option. So clearly that will be deployed for multiple orginsations. Otherwise they wouldn't have created a default node that size.
I would probably send you back to where ever you came from and tell you to re-engineer that. Cloud or no cloud.
Sometimes you need to spend lots of money on tech debt. I think it's nuts that was required but it was.
I wish they’d bring those same prices to some US data centers.
Take a look at their datacenter in Germany: https://www.youtube.com/watch?v=5eo8nz_niiM
Love how they are totally not ashamed to kick off the video with their collection of 14,000 mini-tower desktop PCs. Not rackmounted. Mini-towers.
Also totally ultra-curious about the PS/2 kvm. All those machines are from an era when USB keyboards had been around a long time already. Wondering if this is a security measure...
1, when they need to adjust rapidly between different resource usage profiles, e.g. because they are growing rapidly and can't predict what the usage will be X days in advance
2. They have huge resource requirements and don't care to invest in their own infrastructure, but can negotiate lower rates with a cloud provider
3. When their resource usage is modest but profitability is high enough that cloud expenditure is a rounding error
One can add new servers in minutes, removing has a bit more latency to it, but I'd figure with the huge price difference between rented and cloud you'll come out on top with the former in most case. Also, just use a clustering or orchestration layer in between, they range from very simple to setup and use (e.g., Proxmox VE), to quite complex but also very capable (OpenShift, kubernetes, ...).
> 2. They have huge resource requirements and don't care to invest in their own infrastructure, but can negotiate lower rates with a cloud provider
Using hetzner or other providers is not investing in their own infra, that's using (= renting) the providers infra and ability (peering, fast uplinks, datacenter perks like utility redundancy and staff on site). The second sentence may be true but probably not for most use cases that aren't huge yet, like the post here.
> 3. When their resource usage is modest but profitability is high enough that cloud expenditure is a rounding error
IFF, yes, and often infra costs are relatively low compared to salary costs, so that's definitively some optimization problem one should go through when deciding such things. Chances are that for most projects the profitability can be good but not magic money printing and infra costs are a non-negligible part that eats on their revenue, and then it's definitively worthwhile to think about avoiding the high premium most of those cloud offerings ask for.
With Vercel I don't ever think about adding servers at all, huge win.
> infra costs are relatively low compared to salary costs
Enterprise SaaS here, this is it. Any second my team spends not caring about infra is well worth it.
Also, as you can see in a screenshot on TFA: Some services are simply dirt cheap. The storage account and its various “sub-services” is such a thing. It’s hard to compete with dedicated hardware here.
Depending on your dedicated hosting provider, the traffic cost trap exists, too. Hetzner is a bit of a special case.
Hetzner also offers managed servers where all this is taken care of, for relatively fair prices.
These things are now trivial enough that it doesn't make sense to pay 10x the cost of bare metal for a cloud provider to solve them for you unless you have a crazy amount of runway or absolutely no idea what you're doing.
I've been running something on AppEngine for 10 years and it costs me less than $1 a month. Not sure I could find a cheaper VPS.
On the other hand, I also manage a Mediawiki install, and a cheap Hetzner VPS works great for this.
apt install unattended-upgrades. And Hetzner's firewall.
Typical FUD. On modern servers and the type of software it occupies very little time. You'd spend more managing your cloud architecture.
Cloud providers love it when people do this and are famously easy to talk to when you get an unexpected invoice high enough to require remortgaging your house to even begin addressing it, but I think unless you're working on a side hustle that inherently will need to run in the cloud regardless of scale or are experimenting with cloud technologies in an explicitly time boxed toy project, using cloud services is the financial equivalent of handing a hobbyist craftsperson one of these chainsaw angle grinder attachments that even professionals find hard to keep from bouncing into your body.
If you do want to use cloud services for anything you pay out of your own pocket, the first consideration should be cost management and monitoring. Your employer might have big enough pockets to shrug off a runaway compute instance you forgot about for a month, but that can quickly translate into money that can be anything from inconvenient to life altering if it comes out of your personal budget.
Or just stick with the free tier and make sure everything simply shuts down if you run out. Sure, a "bandwidth exceeded" error page might not get you as many upvotes on HN, Reddit or social media, but it also won't impair your finances.
Of course, the delayed sticker shock is a problem.. I think Google cloud actually lets you create a budget that turns services off if they go over, so there's a solution here if you run a hobby project that you suspect might take off and cost you more than it's worth.
Hetzner has nice dedicated servers for €33/mo:
I'm on an older one that just got bumped up to €29/mo due to increasing electricity prices; it was €21/mo until now, and I can't blame them for that one. The specs are E3-1245 V2 / 16GB / 2x3T, there's over 45 vhosts on it across ~25 wwwroots plus other random services, and CPU usage is basically nothing. The cores are really there just to handle bursty stuff. Most random side projects and small websites don't need almost any resources on modern hardware.
Previously I was on a Scaleway Dedibox, which go as low as €15/mo right now. It was €10 at one point even.
But the ISPs I know do not bill $$$ if you use the max bandwidth (max bandwith they did advertize to you btw) for a sustained amount of time: they'll just start throttling you.
Anyway GP ain't asking about "cloud vs hosting at home" but about "cloud vs dedicated server(s)".
Their monthly cost is something between 0 and a few cents.
Stuff like Hertzner is fine, but if you know your way around AWS you realize have massive cost savings. Prob the same for Azure.
Finally, in many places 40 EUR for a pet project is actually a lot of money.
For ~4€/month (depending on your country), Hetzner offers "Hetzner Cloud" servers with 2GB of RAM, see https://www.hetzner.com/cloud?country=us
Hetzner has such a VPS offering (2GB RAM/20GB nvme SSD/20TB bandwidth), netcup has one for ~3€, contabo has 8GB/50GB nvme/32TB for 6€/month and there are plenty of smaller companies around the world offering similar deals (usually somewhat less included bandwidth outside europe though).
EDIT: Personally I pay 9-10 EUR per month to Scaleway for a 2G RAM and 2 CPU VPS, private docker repo and S3-compatible storage which holds data and some backups, which run both my personal services and some toy projects when needed. I am not affiliated with them in any way
As for costs, setting up billing and usage alarms on AWS is absolutely trivial.
Finally, using stuff like S3 or dynamo for storage gives me a peace of mind I will never have when managing my own servers.
Doesn't change the equation, unless you set up all your PAYG cloud infrastructure and never use it.
A month later a ntp security vulnerability was discovered, soon the server was put offline, some 'patch your things asap' not so nice emails came in. From that time my take is one should spend some time probably daily on an own server if one wants to mantain it.
> A node is an Azure virtual machine (VM) or cloud service VM
> The terms node and VM are used interchangeably occasionally
> Azure Batch creates and manages a pool of compute nodes (virtual machines)
> In an Azure Batch workflow, a compute node (or node) is a virtual machine that processes a portion of your application's workload
So no, seems Azure Compute Nodes are VMs, not bare metal.
I use the credit card of my employer. For my own projects I use my own server for everything. Granted, it doesn't get much traffic.
Some offers from cloud providers are pretty good. If you want to scale to more (virtual) machines, it can be more easily done with the usual providers. I also expect Amazon to know more about firewall and reverse proxy configuration, it renews my certificates automatically and has rudimentary services for monitoring of server state. There is a certain convenience to it.
Would I recommend cloud based hosting? Absolutely not. You become dependent on the provider and prices are often steep. Even if you do not know much about server security, your unsecured s3 bucket will be far more exposed than your standard db installation on your own server. Better build expertise for systems you have full control over than to invest the time on the details of AWS which are more subjected to change.
Well that's the first issue. Many people have automated large parts of their infrastructure in this way so that distributing one huge file becomes part of that whole mess. The goal is of course to keep costs down to a minimum. You can actually do a lot with little money using cloud services.
But the careful balance is that you can easily miss little details. But how does that differ from any systems administration? The details are just in new areas that didn't exist 5-10 years ago.
And the details you miss are more likely to increase cost. And when you process a lot of traffic, you're popular, that can go real fast.
20 years ago in hosting we might get a porn stash on a hacked NT4 server that would draw bandwidth. And back then a whole company might have 100Mbit fiber so you'd notice.
I really don't understand the cloud craze. Everything is more complex to debug, more expensive, and more shitty in all the possible ways you can imagine. I mean i was not exactly a fan of the VPS craze 10-15 years ago, but at least it wouldn't automatically ruin your bank account whenever you got a little traffic.
Kudos to the author for having so much money (thousands in one month?!) to waste. I wish i did too :)
Also, the cloud offers managed software as a service. You don't have to manage your own HA DB cluster or PubSub. It's all just there and it works. That can save you a lot on technical labor costs.
But yes, I do agree with your point. If you don't know what you're doing, you can nuke your budget super quick.
"This is the cost of scaling, this is the cost of owning our own infra, how does that fit into our budgeting and requirements?"
There are plenty of tools and systems that can present a sufficiently linear cost relationship to load and usage that, should your COGS versus revenue make sense, the marginal cost of increased cloud resources a no-brainer--especially versus always-paid-for hardware. If you don't have such a linear relationship you're as much in the position of deciding whether the project is viable as you are anything else.
And even if you do, which I think you’ll agree Troy Hunt does.
If I were building a new business, I would use both cloud and colo. But I do understand that everyone don't have that luxury.
The lead time for Hetzner or OVH is measured in minutes and is appropriate for the majority of use cases. Old-school providers like these used to run the entire internet before the cloud craze started.
Colocation & own datacenter is not a good choice for most startups. Seems like a lot of people here miss a step and only go from one extreme to the other. There's a middle-ground of renting bare-metal that gives you amazing price/performance with none of the drawbacks of colocation or running your own DC.
As to difficulty, they “solve” organizational problems by avoiding sticker shock when someone wants 100+k in equip that’s often a huge number of hoops to jump through and possibly months of delays, a giant bill every month and nobody a complains about the electric bill etc.
you can rest assured that even the largest company will come looking for the person responsible for increasing an expense by that large of a percentage. So maybe it doesn't come out of your personal checking account but you will certainly pay for it.
It’s easy to justify having a larger bill with more traffic. “A retail store isn’t going to complain that they need to buy more stock after selling more stuff that’s just a cost of doing business.” Meanwhile it can be hard to justify a capital expenditure just because traffic increased.
As a case in point, I worked in standing up a critical system in a large enterprise a few years ago. We spent about $12M on compute, storage, networking, etc. At operational state, it was about 40% cheaper than AWS. The problem is, it all sat there for 6-18 months filling up before we fully hit that state.
With a cloud provider, you pay a high unit cost but if you engineer intelligently your costs should move with utilization. Except for government, most entities generally want to see opex move with revenue and prefer to minimize capex where possible.
Coming from traditional infrastructure and development methods, you're mostly right. Part of the expectation of the cloud is that you do things _their way_. And even then each cloud provider does things a little differently. However, if you're willing to subscribe to the <insert provider> way of doing things it (and you'll have to trust me here) makes many things easier. Here's a short list:
* networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up.
* object storage is so much easier than any file hosting scheme you can come up with
* the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward
* I turn all my dev/test servers off at night and they don't cost me a thing
* consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this)
* some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself.
This all being said I'm not advocating an all-in approach without thinking it through, but to do so where it's easy and makes sense.
EDIT: clarity
Bare metal hosts set up the network for you. You may need to know how to configure a local network interface. Even if you actually rack and stack many colos will give you a drop with network set up. You don't need to do what you describe unless you are building your own DC.
> object storage is so much easier than any file hosting scheme you can come up with
That matters if your data volume is truly massive. Only a small percentage have this problem. Also AWS inbound is free so you could upload big data to AWS and warehouse it there if you wanted. Not using big cloud for everything doesn't mean you can't use it for anything.
> the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward
This is the one spot where admittedly you will have to spend more in administration. You'll need to either run your own k8s or Nomad or adopt a different configuration, and you may have to think about it a bit more.
> I turn all my dev/test servers off at night and they don't cost me a thing
You could still do this. Just host live somewhere else. You could also test on a local VM, which is what we do. Obviously that depends on how big your app is.
> consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this)
You only see the savings if they are passed onto you. What we've seen is that Moore's Law savings have not been passed on by cloud providers. Look at what you can get at a bare metal host compared to how much the same compute costs in cloud. Years ago the difference would not have been so large.
Bandwidth costs in cloud are insane, and most use asymmetric pricing where inbound bandwidth is free. This is known as "roach motel pricing" for a reason. Data goes in, but it doesn't come out.
> some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself.
Fair, but they make their money back elsewhere. Those are lures to get you locked in so you now have to pay their crazy compute and bandwidth egress charges.
Here's an example. There are more.
And yeah I agree about the "some services are super low cost so you get hooked" thing. Always been my impression of Amazon: they look for what they can apply scale savings on (usually object storage, it seems) and make it cheap and then over-charge for almost everything else.
The funny business in Amazon's pricing is their Egress Bandwidth, everything is rational.
You're looking at the pricing from a 'cost plus' perspective which is not generally how things are priced.
AWS core use case is IT departments being able to offload all of their infra.
It's a massive, massive advantage. It's so, so much easier and more flexible to use AWS that there is no comparison. It's a 'no brainer' from a cost perspective, which is why, cost usually isn't a barrier with AWS.
Cost only becomes a primary issue when the margin of AWS services is reflected in the cost of the product itself, i.e. when you are hosting a lot of content.
So if you are Phizer, and your IT department uses AWS, the cost is irrelevant.
If you are Dropbox, selling storage for $X/Gigabyte, and your competitors are reducing their prices and you're giving all of your margin to AWS, then you have to do something, i.e. 'make your own infra'.
Is that still cheaper? When you have 30+ very well-paid dedicated DevOps specialists? Maybe it is, I am just skeptical while looking at it as an outsider and without solid data.
I bet it still costs you more than my Hetzner one despite me not having to care about turning it on or off. I mean it's great that the cloud gives you this flexibility but you wouldn't need it to begin with if it wasn't so expensive.
Startups growing fast are the secondary audience.
The primary audience is large enterprises where their internal IT costs <<more>> than the cloud costs. Plus internal IT provides those resources after 6 months...
In what universe? This frictionless perfect vacuum where traffic comes in a wholly predictable consistent continuum?
Also, we may be taking the problem the wrong way around: do these multigig files need to be accessed by everyone from a web browser? No, it's a dump file used by specific people in specific circumstances. Then why are we using HTTP for this in the first place? In this case, only publishing over Bittorrent/IPFS makes sense and many people will happily seed, pushing costs toward 0 for the publisher (and very close to 0 if you only push to a first circle of mirrors who can then push to more mirrors, some of which can be declared webseeds in your torrent).
The opposite, I don't understand why anyone would ever put up a server if they didn't have to.
It's not 'processing power' that's going to be the 'big cost' for most projects.
It's headcount and salary.
If you can materially improve the operating ability of your company, then a few $K in cloud fees is dirt cheap.
I used to work at a 'tech company' that made a physical product and our IT was abysmal. We had to wait weeks for our sysadmins to order blades, get things set up, there were outages etc..
If a project is definitely going to be 'a few linux servers and never more' - even then it would be cheaper and more reasonable to use virtual instances.
The time to 'roll your own' is when the infra. operating costs are a material part of your business.
For example, 'Dropbox' invariably had to roll their own infra, that was inevitable.
Similarly others.
That said - as this article indicates, it's easy to 'over do it' and end up in ridiculous amounts of complexity.
The Amazon IAM security model has always been bizarre and confusing, and the number of AWS services is mind-boggling.
But the core case of EC2+S3 +Networking, and then maybe a couple of other enhanced services for special case works fine.
I also object to what I think is a vast overuse of Cloudflare, I just don't believe that in most scenarios needing to have content at the edge really changes the experience that much.
This only really applies to fully-managed services such as Heroku. Every other cloud still needs a DevOps person according to my experience in many companies.
Just security alone, in terms of managing access to all of those resources, various forms of backup, across regions ... it's just out of reach for most organizations.
This only happens when consumers fail to set budget alerts. Troy could have saved himself $10k with 15min worth of work.
Current workplace is considering a fully self-hosted stack as a unique selling point for the customers and segments we're in. That means, we have storage and linux admins available, as well as tooling and know-how how to run this securely and efficiently. Thus, placing large and often downloaded files on our file stores at hetzner is very much a no-brainer, because it adds very little workload to the teams maintaining these stores and it's cheap.
However, this can be a daunting thing if you don't have this skillset in the org. It can be learned, but that's time spent not working on the product (and it's not trivial to learn good administrative practices from the hell that google results can be). At such a point, a cloud service just costs you less man-hours. And again - it wouldn't be much time for me, but it would be a lot of time if you had to figure all of that out on the fly. That's essentially why the saying goes that cloud services save you time, but cost money.
This recurring question of "why AWS/Azure instead of Hetzner/OVH ?" keeps happening because people are incorrectly comparing higher-level PaaS to lower-level IaaS without realizing it.
PaaS != IaaS are not equivalent. IaaS is not a direct drop-in replacement for PaaS to save money if the workload is using PaaS features that IaaS does not include.
The author Troy Hunt is using the higher-level Azure services like Table Storage (like AWS DynamoDB/SimpleDB) and Azure Functions (like AWS Lambda), and others. E.g. One of the article's hyperlinks talks about using Azure Functions.[1]
If he used Hetzner, he'd have to reinvent the Azure services stack with open-source projects (some of which are buggy and immature) and expend extra sysadmin/programming work for something that's not as integrated. The Azure/AWS stack includes many desirable housekeeping tools such as provisioning, monitoring, routing, etc which he'd also have to re-invent.
TLDR: People choose Azure/AWS because it has more features out of the box. You just have to figure out on a case-by-case basis if the PaaS value-add makes financial sense for your particular workload.
EDIT to downvoters: if Hetzner actually has built-in equivalents to AWS Lambda and DynamoDB, please reply with a correction because I don't want to spread misinformation.
[1] https://www.troyhunt.com/serverless-to-the-max-doing-big-thi...
- My house is probably going to be build much faster if it's built by professional house builder (even more true for services since it's available immediately)
- I have better things to do than building houses
Hum, no. People are asking what kind of value that platform adds that can justify all that risk.
And nobody is giving any clear answer, so I'll stand with my previous answer of "none".
- Security Information and Event Management - exports, alerts, OS configuration
- OS/Application Hardening - Encryption, Password/keys rotation, CIS/other baselines, Drift Management
- Backup - Encryption, (don't forget your passwords/keys are changing), retention, data protection compliance, monitoring, alerting, test days
- High Availability - replication, synchronisation, monitoring, alerts, test days
This is just the tip of the ice berg, if you operate in an environment where Insurance, Reputation, Regulatory Compliance, certification, etc.. are important, then it's easy to see why PAAS solutions are desirable.
AWS is cool and all and has a bunch of interesting stuff, it's just expensive.
From the article:
This was about AU$350 a day for a month… priced at AU$0.014 per GB
A company could not stay in business if every one of their “unlimited 1 Gbps” customers for €40 per month actually used that bandwidth.
I use rented dedicated servers for everything, and always travel by bicycle or transit. It's not as ridiculous as you make it seem.
You might like installing and configuring software, I don't. I'm more than capable of doing so myself, but I'd rather build things on top of other things. I'd rather use a battle tested Secrets Manager and have db replication set up for me. I'm grateful to people that like doing these things I don't and I'm expressing my gratitude by contributing to their paychecks via my cloud bill.
To go back to my initial reply, if you change the context, eg the context is driving a car, you can't possible crash the car you are not driving. If the context is "get home after a few too many pints at the pub", then riding a horse is much better than driving a car (and crashing it). Context.
The reason to use cloud-style services is so you can focus on building the product quickly instead of building and maintaining architecture. But once the product is stable, a cost-reduction pass is in order.
A lot of cloud cost objections would be solved if they defaulted to that instead of defaulted to just charging you the fees. That has its own tradeoffs, of course, but I find myself suspicious that the reason the clouds work this way isn't so much a cold and sober consideration of the aforementioned tradeoffs so much as "this way makes more money when we charge people lots of money they weren't expecting" and "this way makes lots of money when the people deploying the service are organizationally and fiscally disconnected from the people paying for it so they care and notice less".
>> why anyone would sign up
It happens more often than you think: people sign up for credit cards and forget to pay the monthly bill in full. Sign up for a cell phone plan and get charged with large bills of international roaming. People sign up for monthly subscriptions, and exceed the usage limits.
He has a writeup here on how he gets costs down in a big way: https://www.troyhunt.com/serverless-to-the-max-doing-big-thi...
The sum of the GB figured shown in the OP doesn't even amount to 200GB AFAICT. But even if it's something like 10TB that's still not super expensive on many hosting providers.
($350 per day at .014 per GB)
> It's costing me 2.6c per day to support 141M monthly queries of 517M records.
Also, you might be able to store 1TB of data on a spinning disk with no problem but can you run the amount of queries he needs? Will you be able to run them as fast as you need? How much RAM would you need? etc.
To handle that day of getting 1 million customers, which you've been forever optimising for.
Any.. day.. now...