AWS free tier data transfer expansion
aws.amazon.com
aws.amazon.com
https://blog.cloudflare.com/aws-egregious-egress/
https://www.cloudflare.com/press-releases/2021/cloudflare-an...
For an honest bandwidth offer, I'd much sooner consider Fly.io rather than Cloudflare. At least with Fly their true pricing is transparent
edit: speaking of transparency, https://imgur.com/HnlWFUe
Granted Cloudflare, the CDN, has Enterprise plans for higher TB bandwidth (esp video), but Cloudflare, the Cloud platform, has more than generous free-tier, batteries included. AWS' value-based pricing has them extract fees for things as trivial as builds and deploys, and their bills are nothing but nightmare to parse or estimate. This is in stark contrast to the simple and straight-forward pricing with Cloudflare, which we pretty much prefer as a small dev shop. So much so that we choose to pay Cloudflare money to host our services even though we've got 5-digit AWS$ credits.
I'd always prefer paying for certainty than design a solution built on a lottery.
Using any "free" service is generally not free as you scale. That's the freemium model we live in today.
> Also like Google, they're a public company nowadays and will eventually succumb like every company before them to the realities of reporting growth.
This is an unfortunate assumption with nothing to go on at this point. There is no more certainty with AWS, as implied in your statement, than with any other cloud provider. Not all organizations have an end goal in being the scale of AWS. And not all organizations put profit over product with respect to an outdated perspective that said organizations need to grow 40% YoY for all of eternity to be successful. It's now, more than ever, very clear that AWS profit margins on data transfer are egregious and they spin the backpedal as "Oh - look at us dropping prices, for you, our esteemed customer!". This is the real marketing slight of hand here, not the other way around.
I have been screwed, personally, by enough "free" and "unlimited" offerings to never believe them.
On AWS, all the price changes I've had have been to reduce my costs. This is over a pretty long period.
So inform us of the uncertainty with AWS.
Google, sure, they could cancel or 3x your bill (hi Maps API customers etc). AWS does not have that history.
Cloudflare has secret pricing - that's the really annoying thing. Seriously, put a porn site up online with cloudflare and see how far "free" gets you.
There was never any mention of "volatile" pricing. Egregious? Yes. Volatile? No. There's a significant difference of meaning with those words.
Here's a perfect example [0] by Corey Quinn.
> So inform us of the uncertainty with AWS.
I didn't mention "uncertainty with AWS". I mentioned that there is no more certainty with AWS than with any other major cloud provider with respect to your statement about public companies who "eventually succumb like every company before them to the realities of reporting growth". And then for some reason you pivoted to your own, personal, AWS bill from there. I'm not exactly following the logic.
> Cloudflare has secret pricing - that's the really annoying thing.
At this point I'm not sure if your comment is even serious. First of all, please elaborate on "secret pricing". Sounds like serious charges we should all be aware of. Maybe it's with the article from 2019 on The Register about domain pricing? That's not exactly in the context of this thread, but please enlighten the masses.
> Seriously, put a porn site up online with cloudflare and see how far "free" gets you.
I'd charge you with the same ask on AWS. You seem to imply the "free" tier, on AWS, will provide proper capabilities to host an adult content site. I have strong doubts about this. The logic of this argument is ill conceived at best. Or is your logic just that you can't do this on Cloudflare and that's the root of your argument on why AWS is better? Again, I'm not exactly following your train of thought.
[0] https://www.lastweekinaws.com/blog/the-compelling-economics-...
Can you say the same about cloudflare? No. Can you say the same about oracle? No - they have a miserable history of screwing customers.
AWS is offering clear pricing, cloudflare is not. It's really that simple.
This makes me realize that folks just don't understand the value AWS is providing, and is perhaps why they can charge such insane prices.
People with actual money to spend don't want "free" because they don't believe it's actually free.
In terms of cloudflare, they have something like a negative 60% operating margin. The idea of building a business on a company with a negative 60%+ operating margin is insane, either they will go bust or have to raise prices.
AWS by contrast makes money. Because of this, they can shave a point or two off margin to give (another) price reduction.
1TB per month cloudfront, 2M cloudfront functions etc etc. They are under almost NO financial pressure to raise rates.
Cloudflare is under pressure or will be. With VC money perhaps they will get a longer runway.
The "free" offerings are an old story by now.
It's also fantastic to read the misconceptions of the "value" AWS is providing. In some cases they do provide a much greater value over cost ratio, but if you've convinced yourself that blindly for AWS proper as a whole - boy do I feel for you the day you realize the economic advantage they manipulate to monopolize the cloud market, and not for the greater good of their customers.
Clear pricing you say? I'll use Corey Quinn (The Duckbill Group [0]) as an example again - his entire liveihood and business runs on the fact that AWS pricing is not even remotely clear. It's laughable that anyone would publicly make that statement at this point in time knowing what we know. Sure, if you're running a static site on S3 for a few users a month I'm sure you've got it covered. For those of us dealing with large scale enterprise everything stated here is, at best, bending the truth and at worst flat out ignorance.
Corey's business exists because prevailing engineering culture encourages pretty much the entire industry to consider optimization as an afterthought, not because engineers can't understand AWS pricing, or interpret a few bar charts in Cost Explorer, and in the face of a deadline, if it's not on the agile board everyone knows it doesn't exist.
Many of Corey's technical posts are around finding sweet technical substitutes for niche use cases, but as I'm sure he'll tell you, 80% of what he does is easily discovered a few clicks away from the AWS home page.
> as they build out their pivot towards targeting the enterprise market
Cloudflare have a solid sales pipeline, but they're a sitting duck if any of the big clouds ever decide to replicate the business model like-for-like. One of the reasons this may not have happened yet is because Cloudflare's whole presentation is consumer oriented, starting with domain configuration management that is hell to version correctly when 20 people have access to the account. Outside some sweet Javascript cold start hacks they basically have no moat, and there are far more situations that could send the company into desperate measures than otherwise.
Nobody is actively encouraging the entire industry to consider optimization as an afterthought. This makes zero sense. Why would an organization pay a business to reduce their AWS costs if it wasn't worth the cost to realize the savings? Cost optimization is an easy task, as you've stated - so the cost/value proposition of a business like The Duckbill Group must not be worth it according to your statement. Yet they exist and do, seemingly, well. Maybe... Just maybe, cost optimization in AWS is not easy, not straightforward, and designed to be painful enough to where smart engineers are incented to leverage Amazon's dark patterns of hiding costs at time of deployment.
You even state...
> 80% of what he does is easily discovered a few clicks away from the AWS home page.
So why is cost optimization a constant point of conversation with AWS if it's so easy? Why do outfits like Digital Ocean advertise on the notion of clear billing as a positive differentiator compared to AWS?
> Cloudflare have a solid sales pipeline, but they're a sitting duck if any of the big clouds ever decide to replicate the business model like-for-like.
Let's take a stroll back in time. Do you think that Amazon and AWS have always posted a profit? Go look, they've posted many quarterly losses to get where they're at. That's how it works as you build a business like that. To your point - AWS does compete directly with Cloudflare in certain products, yet here we are, Cloudflare and AWS both continue to grow (negative operating income / net income are not a direct correlation of company growth BTW). A mistake you've made is around brand and reputation. Nobody thinks of AWS as a security company. Customers continue to buy Palo Alto Networks, Fortinet, Zscaler and, yes, Cloudflare - even though AWS offers some overlapping portfolio. Why? AWS isn't viewed as a security portfolio. Cloudflare has brand reputation in security and content distribution. And it's a pivot that easily works with both their brand and reputation.
> Outside some sweet Javascript cold start hacks they basically have no moat, and there are far more situations that could send the company into desperate measures than otherwise.
This just screams of the competitive argument low-road. I don't have ties to Amazon or AWS. I'm not an employee. When I read statements like this it's affirmation that there's some agenda. I laughed out loud reading that as the closing argument, thanks for that.
Limited IO bandwidth in middle and upper management alongside difficult schedules (we covered that one already). Take 2 steps above engineer on an org chart and detail becomes invisible, the vast majority of tasks begin to resemble a teenager at a mall with their dad's credit card. Meaningful technical validation phases are almost unheard of in many organizations, and largely antithetical to agile.
> Why do outfits like Digital Ocean advertise on the notion of clear billing as a positive differentiator compared to AWS?
Because they market to folk who never take the time to model comparative costs. In any project I considered them for (3 I think, <$15k/year each), Digital Ocean was significantly more expensive than AWS. I follow my spreadsheets, the industry follows marketing.
> Nobody thinks of AWS as a security company
They're the only vendor I deal with who are on first name terms with the NSA and sell in tremendous quantities to the US government. CloudFlare on the other hand, to this day, default to MITMing SSL connections for new accounts and downgrading them to cleartext en route to the back end. It seems our perceptions differ wildly.
I'm sorry, but what are you trying to say? "Antithetical to agile" - is there a point beyond some non-nonsensical, made up scenario? Every major organization with a cloud budget cares about optimization today at some level. This hasn't changed in over 20 years because those budget dollars used to be directed at data center costs. Now they're more fluid and can be more impactful when people make mistakes or aren't making sure to optimize up front.
> Because they market to folk who never take the time to model comparative costs. In any project I considered them for (3 I think, <$15k/year each), Digital Ocean was significantly more expensive than AWS. I follow my spreadsheets, the industry follows marketing.
Then share some examples from said "spreadsheets". Because for straight instance pricing DO beats AWS pricing in most every way. This is one of a handful of bread and butter services AWS offers (lift and shift compute). DO also does, most often, better with respect to performance (CPU/compute) when comparing directly [0][1][2][3]. This calculator [4] at DO showcases cost comparisons across all major cloud vendors and I've validated comparison in the last month with AWS - the prices check out.
> They're the only vendor I deal with who are on first name terms with the NSA and sell in tremendous quantities to the US government.
This is a rather naive comment. I happen to work in the security industry and every cloud vendor has direct ties to 3 letter agencies - that's not at all unique to AWS. Sorry to burst your bubble, but also every notable security player in the industry has similar relationships. It's not unique. It's also more advantageous for the three letter agencies in many of those relationships. Also, AWS, along with everyone else - has relationships in security information sharing for verticals. So, yes, AWS is part of FS-ISAC, as one example. Just. Like. Everyone. Else.
[0] https://www.vpsbenchmarks.com/compare/docean_vs_ec2 [1] https://www.bunnyshell.com/blog/aws-google-cloud-azure-digit... [2] https://www.digitalocean.com/resources/cloud-performance-rep... [3] https://www.upguard.com/blog/digitalocean-vs-aws [4] https://www.digitalocean.com/pricing/calculator/
I'm saying this is my bread and butter, and I've seen the same pattern in every company I've crossed the doors of. It's no made up scenario when a team of contractors have spent 12 months building out some service, not a single person will have given a damn about costs, so cost optimization is purchased separately. The contractors aren't wrong nor is the business wrong, nor is this pattern unique to cloud spend.
> I happen to work in the security industry
Then surely you will understand how an entire subindustry can exist to mop up after engineers who could otherwise 'simply' avoid most mistakes if they just spent more time Googling.
Yes, I'm sure an entire $50 billion dollar industry comes down to, like all your arguments, just lazy people and the simple fix is "Googling".
You make valid points, but what you're missing is the same accusations you level at Cloudflare were once leveled at AWS (which is why AWS had virtually no credible competition from Y!, MSFT, GOOG for 7 years!).
Also, Cloudflare's moat isn't 0ms cold-starts, but their persistence in commoditising bandwidth. Think Amazon Prime free 2-day shipping and how that worked out...
If you don't think AWS bandwidth pricing is something in the real world I don't know what to say :) AWS is now broken out in Amazon financials - worth a look to see what they are raking in and the margins they are getting in the real world :)
"Clear pricing you say? I'll use Corey Quinn (The Duckbill Group [0]) as an example again"
His entire business depends on the fact that AWS pricing is clear and public. For a service like cloudflare (ie, call for pricing / dealing with a salesperson trying to figure out how much they can squeeze you for on upfront or on renewal) this type of service is much harder.
In short, if your bill is too high, you can talk to someone like Corey and they can probably help you bring it down.
This comment confirms that you have a misunderstanding of how organizations can and have bought bandwidth via high-cap Internet offerings in the real world given how you've misunderstood my argument completely. My point is that organizations like AWS, Cloudflare, GCP, and many, many organizations around the world still buy Internet connectivity this way (directly). My unchanged statement, all along, has been regarding the egregious margins AWS makes on bandwidth that they charge for in a very asymmetric and oversubscribed manner. It's hard to have an conversation about these things when the basis for how businesses operate aren't understood by those making comments, unfortunately.
> His entire business depends on the fact that AWS pricing is clear and public.
I'm not sure how many enterprise agreements you've helped derive or review, but you can get agreed pricing from any cloud vendor - Cloudflare included. If you're spending any amount monthly beyond what appears to be a personal account this is not your issue.
I'm still using free tier Google Apps in multiple places, even though they haven't offered new free accounts for about ten years now. They even still let you create new users for free in these legacy GApps.
Interestingly, I would likely migrate off of GApps to a different paid service if Google changed their minds, however I don't think they have a strong incentive to apply pressure here at long as Gmail.com accounts are free.
That's huge, Could you explain a bit more on how you achieve that? I've been exploring Cloudflare cache, AFAIK for anything not cached by default you need a special page rule e.g. for .html, I tried a wildcard domain.com/* and it doesn't seem to make any difference. Are you using workers to cache specific file types?
Yes: Using Workers as an api gateway and the Cache API as CDN.
[0] https://developers.cloudflare.com/workers/learning/how-the-c...
Even within just the CDN category, Cloudflare does a lot with its basic CDN offering (bot blocking, transformations, dynamic caching, etc.) that other vendors may charge as separate options.
That's not to say I don't believe you, I just wanted to make sure it's apples to apples.
Cloudflare's pricing model only really makes sense, IMO, if you're either a small/med business (which makes it an incredible deal) or if you're working cloud-native using their edge functions (workers, pages, etc.). If you're primarily using them to shield and proxy a LAMP monolith or similar for a large number of users, yeah, I can see how that would get expensive really quickly. There are other vendors who specialize only in that, being a dumb CDN. Cloudflare's value is that they enable completely new architectures based on their network topology... you can't easily do something like that on Fastly, for example.
On the other hand, $0.0021/GB is far on the cheaper end of the spectrum, who is it that offers something that low?
Agreed all around that the pricing is frustratingly opaque.
fly.io looks interesting, though, never heard of them.
It's a test account, I just want it to shut down when the limit is reached.
https://aws.amazon.com/blogs/aws-cloud-financial-management/...
Should _every_ S3 action have a `if will_incur_charges() and should_not_incur_charges(): raise Exception()` statement in it's critical path? No, of course not. Everyone get's slower for nobodies benefit. It has to be delayed.
But then you run into an issue: what if you end up costing AWS 100$ before the budget action kicks in. Should you not pay that? Why not?
My strong guess is if you had a free account, setup a budget cap, went over it, and they decided to charge you*, a quick email to support would get it waived.
I’m very much a fan of AWS, in part because while they have the chance to uphold the legal terms, my experience is that they’re pretty customer friendly.
* Early on, I had many bills under $1/mo that they just comped without me having to do anything.
We're talking about one account-wide flag `has_exceeded_billing_limits`. Changes are infrequent, and can be pushed into caches. Small overruns while the flag pushes are trivially eaten by AWS.
The amazon deal is simple. Very clear pricing for pay what you use.
Cloudflare - can you link to the page where they show the cost of bandwidth? Still waiting.
Budget actions work by applying an Deny All to IAM, which has essentially exactly that.
The problem is not the shutting down, it's the detection. AWS billing has a resolution measured in hours, which has limited usefulness on a platform where you can rack up thousands of dollars in charges in just a few minutes.
So you get an email saying your $10/mo site is now $1000 for this month, and climbing.
In general I wouldn't recommend using AWS and expecting the free tier for anything that's going to be public facing or autoscaling.
It's not a solution at all given the fact the alerting process can lag behind the logging process by several hours or more. If you've hit a traffic spike it could have rolled over your site and gone in that time, leaving you with a bug bill.
Alerts are not a viable solution to traffic spikes unless they're real-time and absolutely bulletproof. AWS's alerting is neither.
For the billing system to then "turn off" X it needs a number of things.
1. It needs the ability to reach back out to that service. It probably has no idea what it is, all the billing system is likely to receive is something like:
{service_name: "X", action: "Put"}
So now your billing system needs to be able to resolve "X" back to some AWS resource that it can talk to. Both the resolution of X as well as the "billing can now talk to every single AWS service" are pretty heavy lifts.
2. It needs to know what "off" is. "Off" for S3 could mean a lot of things.
a) Delete the bucket and all data inside of it
b) Keep the bucket but delete all data inside of it
c) Keep the bucket and the data but disable API access
etc etc. Do you disable PUT? GET? Both? What if what's blowing up your billing is GET?
And this really doesn't get easier for other systems. Do you back up a database before killing it? That incurs charges too.
I don't see AWS somehow solving this in a "one size fits all" way because there isn't one.
It's not an engineering problem at all at its heart.
It's a marketing/business problem that someone somewhere is thinking that Amazon can provide a free service to X users, knowing that Y (X, Y positive, Y << X) users will go over their "free tier" usage and pay for all X's costs, maybe even more, making the free tier a profitable business on its own.
My point is that it is definitely an engineering problem as well as a product problem.
a) It's going to be super technically difficult to build (especially in a way where it's responsive at a granular basis to handle huge blow-up bursts)
b) It's not even clear what you're supposed to be building
None of what people are proposing is well defined or easy to build.
The way you usually solve this is by having sane defaults, and giving users different mechanisms for configuration based on how complex their configuration needs are. This can take a tiered approach.
As an example, simpler and straight forward things (such as disable egress traffic from S3 if the bill exceeds X) can be in the UI itself. Then, for customers who need more control, an option to configure via json or yaml similar to cloud formation. For anyone who needs even more giving an option to call a customer defined lambda function would give them the ability to at any metrics and take appropriate action.
Engineering problems:
1. How do you actually have a billing system reach out to every other system? It has to resolve the resource, network with it, have IAM permissions, a network route, etc.
2. How do you handle consistency?
3. How do you make it responsive?
4. How do you add this to every billable entity?
I mean, it's just a shitload of work, and all of that just to get to a terrible idea.
> The way you usually solve this is by having sane defaults, and giving users different mechanisms for configuration based on how complex their configuration needs are. This can take a tiered approach.
The sane default is you pay for what you use, and you can listen to billing events and build all of the logic you're talking about if you want to.
So yes, it is shady not to protect customers from that IMO.
The complaint here is that Amazon offers a free tier supposedly for learning the platform, but it is a giant footgun that shoots a ton of people in the foot.
People are reasonably asking for hard limits to protect them from this highly foreseeable situation wherein a complicated cloud offering can go on a spending runaway.
It is literally as easy as following a beginner tutorial and selecting the database instance the tutorial uses and leaving it running. That could be a several hundred dollar mistake.
I don't think it's unreasonable to say that if you're using AWS you're taking on some responsibility to make sure you're not blowing up your bill. AFAIK you are automatically enrolled in emails that will tell you when you're about to exit a free tier limit, so it's not like they won't warn you.
Then make it opt-in but a highly visible one during account creation so that people who just want to test can enable it.
It would be excellent for anybody intending to use the free tier for its stated purpose (getting to know the AWS platform) who would like to make the _choice_ to shut off their services if they are going to exceed the free tier quotas.
That way you are free to experiment, and if you blow up something while learning, you're not then leaning on the mercy of AWS support to refund you.
Playing around turns into production.
When your service goes down and you lose $XXXXXX revenue suddenly it's AWS's problem. AWS has taken the approach that keeps the lights on, assuming its customers understand their unit economics.
AWS Free Tier
Gain free, hands-on experience with the AWS platform, products, and services
But the thought that not immediately reacting to an e-mail can cost me a month's salary or more is terrifying. Maybe it'll get waived. Maybe it'll be waived in the form of credit that I can only spend on the product (i.e. for my purposes, not waived). Maybe I'll be stuck with it. The "maybe" is the danger here.
Let's say I put some obscure 50 MB dataset into a public S3 bucket, and pay my <1 cent per month to host it and like a dollar for each 200 downloads. Rarely does a month exceed a dollar or two, all is good.
Then someone builds a poorly made colab that downloads the entire dataset each time it is run, and the colab hits the front page of HN while I'm traveling, and makes it to social media the next day because it shows something funny. And people don't run it just once, they play with the parameters, running it multiple times.
By the time I'm back and see the e-mail, a 10000 USD bill may be waiting for me.
To obscure? How about this one:
The operator of a semi-popular website has decided to hate me. Each page load now contains an <img src=[my image]?t=[timestamp] width=1 height=1> in the header, pointing to the biggest image I'm hosting for my small static website.
Edit: Even better example
I've accidentally left an API key in a git repo that I pushed online. My carefully set up billing alert was deleted, then my account started spun up as many of the most expensive GPU instances as quota would allow and started mining Monero. In this scenario, I think a $10000 bill would be "getting off easy".
(Just to be clear, these are hypothetical scenarios. If anyone knows how various cloud providers would react to those in practice, or if you know that there are countermeasures that would reliably stop them, please do tell!)
Have you checked out AWS console recently?
BTW - I've never heard of the later happening at AWS ever, and I have at other hosting providers.
However, I also think a big problem is that many people on the internet and especially people who try to sell AWS tutorials or learning courses push AWS as some toy that every developer should sign up for on a whim without understanding what they are doing. An AWS account is an industrial-grade tool, it's not a toy, and it should be treated as such. It's like renting a backhoe when you don't even know how to use a shovel yet, and then being surprised when you completely screw up your yard.
Sites like acloudguru that offer ephemeral sandbox AWS accounts are becoming more popular, and people new to AWS should really be steered towards those.
It'd be a huge engineering effort to make something instantaneous--I think the closest thing they have to such a system is whatever they use for rate limiting or IAM.
I'm guessing there's a pretty high overhead to trying to do realtime instead of batching
AWS oopsies suck but I think their billing system is pretty robust compared to lots of usage based billing systems (like, say, utilities)
it's pretty reasonable for them to ask for a CC -- making it too easy to get free compute/bandwidth is opening the door wide for abuse.
..but yeah, everyone wishes they'd have a sane way to halt services if over budget.
but yeah, more competition is always good.
But I still don't run anything important on it or push the limits of the free tier. Oracle doesn't have a good reputation. Also "Oracle Unbreakable Linux" is literally just RHEL rebranded, but it's not a community project and they don't like to acknowledge it so it feels particularly shameless; especially since they are selling "support" for it.
Hosting is one of those things you really ought to spend 10 minutes reading and find out what you are and are not allowed to do. It's basic due dilligence when you're renting someone else's hardware and bandwidth.
You basically get stuck paying lots for Oracle software and that doesn't directly equate to value the software provides
See also: Bryan Cantrill's "lawnmower" talk about what happened to Solaris after the Oracle acquisition. https://youtu.be/-zRN7XLCRhc
[1]: https://old.reddit.com/r/sysadmin/comments/d1ttzp/oracle_is_...
[2]: https://upperedge.com/oracle/top-3-reasons-oracle-java-users...
I was at a point where I just said "screw this, I don't care about the free trial, just give me an account where I'm supposed to pay for everything". For some reason you have to go through the free trial in order to get a regular account.
I didn't even try out the platform once I got my account. They managed to drain all my energy in the sign-up process.
I'd rather go bankrupt by AWS/GCP nefarious egress fees than having to deal with Oracle again. Serves me right for giving them a shot.
I second the recommendation to stay as far away from Oracle as you can, even if their OCI pricing seems incredible.
Providers always have a false positive rate on signup experiences and last time I looked Oracle had provided a way to contact them and say "You got it wrong" (and they are reasonably good at fixing those situations).
Maybe the idea is that if you're doing 1+ TB of CloudFront traffic, you're already deeply locked into AWS anyways and less willing to make the jump anyways.
The cost savings is equivalent to ~$9/mo in standard US regions. Nobody is going to migrate clouds over a $9/mo saving.
If we're talking about CloudFront that comes out to $85. That's actually a pretty good savings, but it's distinct from egress. CloudFront's pricing isn't locking customers in due to egress pricing because it's a CDN, not a data storage service.
With AWS, you can do it, via a trigger on a spend notification and a script, but the whole thing is a giant kludge. It should be a default feature. It should be a default feature for all cloud providers.
I'm literally using them less because this isn't a feature they offer. Even the free tier of AWS is too risky without hard limits.
The unfortunate reality is that hobby developers that just want to pay $20/month aren't the target audience for GCP etc. They don't really care if you're using them less for your personal hobby projects. They target large enterprises, and those large enterprises would have very little use for something like "cap my spend at $20".
Even as an AWS employee, I sometimes use non-AWS hosting providers for my own projects. Even outside of the billing situation, AWS is often too complicated for my use cases. It's just not targeted at me and my hobby development projects.
disclaimer: am AWS employee but the above is my own opinion and not official position of the company, etc etc.
For the enterprise as a whole, probably not. But it would still be useful to be able to create sandbox accounts for experimentation with a hard limit on spend. Or give developers their own cloud accounts to run development and testing infrastructure that they can control, without having to worry about them accidentally spending way too much.
Probably easiest to generate a virtual card number with a spend limit, off the top of my head I know Capital One offers this, Apple will also generate virtual card numbers though I don't know if you can set a spend limit on them.
(I should note that I haven't used them myself, I'm just impressed by their website/business model.)
Imo hard spend limit is a pretty complicated thing to implement and billing has historically been very batch based
It still baffles me that this isn't the default. Then again, egress/ingress costs also baffle me, why not just offer capped speed and unlimited bandwidth?
I doubt that i'll ever willingly use a platform that i pay for myself, which can just decide to charge me bunches of money because of my site/app getting DDoS'ed (assuming not all components are fully protected at the edge or something) or suddenly gaining popularity.
In my eyes, such scalability should be opt in (or at least opt out) and by default my apps should just break under the load, much like any self-hosted software would, which may be preferable in some circumstances (e.g. API manages backpressure as best as it can, once request queues fill up, it just gives "503 Service Unavailable" with something like "Retry-After" and any clients/front ends are supposed to show messages to try again later, or automatically retry after a while).
To that end, here's a list of providers that give you such fixed resources (in this case VPSes) and that i personally have used in the past:
• Time4VPS: https://www.time4vps.com/?affid=5294 (currently hosts most of my sites, hence affiliate link)
• Contabo: https://contabo.com/en/
• Hetzner: https://www.hetzner.com/cloud
• Scaleway: https://www.scaleway.com/en/elements/
• Vultr: https://www.vultr.com/products/cloud-compute/
• DigitalOcean: https://www.digitalocean.com/products/droplets/
Of course, some of those also offer managed services, but in my eyes it's far safer to just run MySQL/MariaDB/PostgreSQL/MongoDB/Redis and any other piece of software in Docker (or OCI compatible) containers, as opposed to using something proprietary, even with protocol compatibility. That way, it's also possible to migrate between providers easily, should the need arise.
This doesn't feel to me like a generous enough statement about all of the finer points.
Would i be okay with my servers no longer responding to web requests (or doing so with reduced port speeds with some of the above hosts, e.g. Time4VPS) in case of spending limits being unexpectedly hit? Yes, definitely, since my projects going dark for a bit or slowing down would be preferable to me ending up not being able to pay my rent and having to rely on public outcry about the large bill and the vendor's mood that day towards letting that one slide.
Would i be okay with my servers running out of space and no longer writing new data to any database, merely responding with the appropriate errors instead? Yes, definitely, because that probably signals the fact that for some reason lots of space has suddenly started getting used for no reason, something that happened so fast that i didn't even get to plan the appropriate scaling, once Zabbix would warn me about 80% of the disk being full (or any monitoring tool would have an alert set up for this). This would probably also be yet another sanity check.
Would i be okay with my servers suddenly ceasing to exist and being wiped from existence, for any reason short of excessive abuse complaints or repeated ToS violations? I most certainly wouldn't want this to happen, yet in my experience it has never actually happened - some of the vendors listed allow you to pre pay for the resources you're about to use (e.g. Time4VPS and Contabo), even offering discounts for longer term reservations much like AWS does, without unexpected charges to you like AWS would do. In contrast, some of the other vendors listed allow you to pay based on hourly usage (Hetzner, Scaleway, Vultr, DigitalOcean, at least IIRC), but also won't have unpredictable pricing spikes because of set limits for the most part (ingress/egress charges may still apply, however, a worrying trend in the industry that muddies the waters), if i pay for a 5$ VPS every month, that's what i can expect to pay most months regardless of usage (assuming 100% uptime).
With these factors in mind, my servers being wiped would probably have to happen due to either me misusing those services, or alternatively me failing to pay the bills even with the more predictable pricing structures, which is actually very much like a person's electricity being turned off due to them not paying for it - as unpleasant as that'd be, no surprises there. Personally, i don't subscribe to the belief that managed services with unpredictable billing are the only way to do software nowadays, something that Richard Stallman coined as SaaSS (Services as a Software Substitute) and about which you can read more here: https://www.gnu.org/philosophy/who-does-that-server-really-s...
Alas, as long VPS uptime with predefined resources is the unit of computation that you're paying for, everything else should be fairly simple from there onwards.
They don't reach into your servers and meddle with your nginx configuration. They don't shrink your disk volumes to the amount that's already in use. That wouldn't be possible.
Exactly, that's why I said that being able to prepay for a month ahead (or a similar period) is a good thing, as is anything else that makes billing predictable (such as no auto scaling).
> No more server time means your servers just vanish into the ether. It doesn't mean they start returning error pages of being unable to write to the database.
Come to think of it, that's probably the case for most people out there who just use one account/card for all of their resources.
What i described is indeed possible with a tiered approach: alotting most of your funds to keeping the DBs alive with all of your data and whatever is left apart from that for the more dynamic components - VPSes that work as load balancers or others that host your APIs.
Of course, the ways to achieve something like that are plentiful, for example, multiple accounts with different virtual cards (which may or may not be allowed), using multiple different service providers for different parts of the system (e.g. if one has better storage plans and latency isn't an issue, for example, data centers in same country), using multiple service providers for redundancy (hard to do for most DBs, easier for container clusters) or even using functionality provided by the platforms themselves (which may or may not exist when it comes to billing, even though DigitalOcean had a pretty lovely way of grouping resources into projects).
Come to think of it, that's probably a space that could use a lot of improvements.
> They don't reach into your servers and meddle with your nginx configuration. They don't shrink your disk volumes to the amount that's already in use. That wouldn't be possible.
This isn't even necessary. If platforms allowed me to say: "Here's a bunch of API nodes that I'll give up to X$ for the following month and here's another DB node that I'll prepay in full with Y$ for the following month" then none of the other multi-cloud deployment strategies or tiering would even need to be considered.
If we want to consider situations with auto scaling or other types of dynamic billing, then we should also be able to say something along the lines of: "For those API nodes with my preconfigured init script, i want at least Z instances available with the given funds, whereas any remainder can be used up for autoscaling up to W total nodes. Thus, if the alotted funds run out, it all will return to the prepaid/reserved minimum of Z instances for the rest of the billing period."
AWS Lambda can charge you for usage at sub second resolutions and yet neither they nor other cloud platforms provide good resource prioritization solutions or fine grained billing limits, doing just alerts at best? I'm not sure why that is, but until things change, we'll just have to live with workarounds. Technically some of that would already be possible with something like Reserved Instances on AWS (https://aws.amazon.com/ec2/pricing/reserved-instances/), but that's still not granular enough IMO. Then you'd just have RIs and "everything else" as opposed to resource groups with spending limits.
Furthermore, if my Time4VPS server runs out of bandwidth, I'm not expected to pay more, the port speed just goes down until the end of the prepaid billing period. That's the sort of simplicity that's one of the best current options, with minimal hassles. And if i ever can't afford to renew 10 API servers, then I'll just get 5 or whatever amount i can afford.
More platforms should be like that, maybe at a day/hour/... resolution, though and with APIs that allow us to decide ourselves how often we want to renew services and for what periods, like a few clever scripts online that you can find for turning off AWS/other service instances when not in use. Currently any fine grained controls that you want on those platforms would have to be done programmatically, but it's not like you could easily manage ingress/egress costs that way.
I'll simply use services that fall more in line with what i want and i urge others to do the same, to better suit their needs and avoid the possibility of some very unpleasant financial circumstances down the road, at least as far as personal projects are concerned. Hence the list of some lovely providers in the post above.
They'll simply continue to work with their current approaches to billing, missing out on the 0.01% (read: humorous made up figure to not create the impression that i believe that huge platforms should work like i expect them to, my ego isn't that huge) or so of additional income that they'd get by catering to all of the people like me in the world. That's the beauty of a (mostly) free market - supply and demand.
I find most apps are pretty binary. Either they are high bandwidth (like video and backups) or not. If you use 1TB there is a good chance you will use 2TB.
This will certainly make it less likely for the low bandwidth users to get a surprise bill but if you're doing video $85 per TB will get expensive fast. You better make sure your business model can make a profit at those prices. But even then, if the average user pays you $20 a month and only streams 100GB of video you make money but you'd make more money outside of AWS.
It's almost as if AWS wants to active discourage those use cases.
I realize business need to make a profit but I don't think AWS is cross-subsidizing. It seems more like there are no loss leaders, they make at least 60% margin on all products (for users not big enough to negotiate rates). They could lower their outbound prices significantly, they just choose not to until the market forces them.
Which is smart business. But I think given Cloud Flare's pressure here, the market is calling for a more aggressive adjustment than just upping the free tier.
[1] https://www.cnbc.com/2021/09/05/how-amazon-web-services-make...
50,000 x 250k = 12.5B
Also, they do cross-subsidize, because many AWS services are either hardly used (CodePipeline) or free (CloudFormation) and the cost to run those services is non-negligible.
Also, since CloudFormation is a feature that facilitates creating more resources that you do pay for I'm not sure that is a great example. That is more like saying that the other services subsidize the Web Console. CloudFormation is not a product in-and-of itself, it is more a shared feature that spans product lines.
Code Pipeline is an interesting one in that if it is true it's not widely used and costs more money than it makes, that's a no brainer: shut it down. But there has got to be more to that story why they haven't.
But in any case, with code pipeline there is a clear value chain that ends at ECR/EC2/Lambda/etc. My guess -- and it is just a guess -- would be that someone feels pipeline produces more revenue for EC2 (or similar) and that covers the cost. Or, simply, they have a path for it to be profitable.
The Corey Quinn number I keep quoting is that >60% of AWS revenue for large (>100MM) accounts is EC2. That jives with what I hear from people in the industry.
That means that the other 250+ services are splitting less than 40% of AWS revenue....
Although their pricing [1] after the first 1TB is still very expensive.
Who would pay 100$ for backup test or recovery with S3?
>100 GB of data per month (up from 1 GB per region)
This only applies to the free tier, which you age out of after a year. Who even cares? Or am I misreading this?
They can do a lot better than this.
edit: It appears that the regional transfer doesn't age out either, even though it isn't explicitly stated in this post (whereas they did state so for Cloudfront).
0: https://aws.amazon.com/free/
Even now you should be getting 1 GB of free out traffic on your older-than-12-months AWS account.
Is it 100 GB per region or 100 GB across regions? Because it sounds like the latter.
But,
This is THE textbook example on how competition pushes innovation forward and drives prices down.
Thank you, Cloudflare!
Is this similar to cloudflare workers?