It's quite a tour de force how Amazon have taken "separation of concerns", applied it to web services and used it to create complex and difficult to understand or predict pricing to print money. Bravo.
The fact that there are consultants who specialize in figuring out AWS billing was, in retrospect, a warning sign.
I wish there were SaaS products that would effortlessly audit enterprise SaaS bills with support for more than just the top 1% SaaS products.
I have a ticket open with MS support. I suppose it's to their credit that they're listening to me but it's been a few weeks with only the aforementioned hand-waving. I started just sending them screenshots showing that their own dashboard showed no usage during the times I'd gotten charged.
AWS you can poke around and figure it out, Azure is a Kafkaesque nightmare of infinitely confusing UI/UX, a permission system that is like trying to hit a moving target in a dark room, being moved by someone that hates you, doing anything seems needlessly complex, and trying to replicate/test anything locally is horrible (looking at you Azure functions).
I'm trapped in this mess because the nature of our Client is that they'd consider anything besides Azure as a slap in the face, but it's been absolutely the most miserable task I've taken on in years.
As a bonus I'm getting side-eyed by teammates who have a naive faith in The Cloud and can't comprehend why this is taking so long and why I can't just give them a cost estimate.
Take this with as much salt as you want but they're really trying to put that in the past. OCI pricing is super transparent, simple metrics, cheaper than the incumbents on every front, there's no "call us for pricing", there's a transparent 30% discount for committed spending (before you even get to talk to a salesperson) and the dev team has been given free reign to act like a startup and ignore the corporate machine. I'm not a marketing person but I do believe we have a very good offering and it's a shame it's being ignored solely because of bad blood.
[1] https://www.oracle.com/cloud/economics/
[2] https://www.oracle.com/webfolder/workload-estimator/index.ht...
You have a company that charges licenses based on the physical core count even if you run the product in a VM using only a single core, but then they also want to be your cloud supplier.
The only strong point of Oracle is legacy product lock in, you would be crazy to pick them for any new initiatives.
Honestly, I wouldn’t touch something from Oracle with a 10 foot pole, sorry. I’m sure the UX is probably pretty good, but I’m staying away from that company if I can avoid it-too many horror stories, way too “enterprise”.
If you feel like you're way overpaying for AWS, you're much likelier to look out for alternatives. So if they help their biggest accounts save some money, it'll net them way more in the long run.
I frequent a local family-owned Italian restaurant/pizza place and my usual order jalapeños, banana peppers, and roasted garlic so I do a build you own. Totally fine, happy to pay the price. But one day I come in and the woman working the take-out counter sees my order before it goes in and tells me that if I order a "meat lovers", remove two toppings and sub the rest it comes out way cheaper.
So now I'm the vegetarian that always gets a meat lovers and she makes the joke every time I come in.
I don't think it's weird or nefarious that a business doesn't have completely consistent pricing and holding it differently can save you money. When we spec out services on AWS we're using the official calculator when deciding whether it's "worth it" so to us at least it genuinely feels like we're saving money.
I've seen that before, and I've also seen the opposite where people buy the veggie pizza and then add a meat or two on top to get a cheaper supreme pizza.
We had originally gone in with clever ways for owners to describe how different changes would affect the price of an item, but ended up landing on just a giant matrix where they could input the end result price for any combination. The owners couldn't articulate their own pricing well enough to model it well.
It's part PR, part long term customer acquisition
* Price things in an intricate way sufficiently specific to actual costs of providing the service for many use cases
* Price things more generally in a way that more or less probably results in about the same revenue
The problem with option #2 is that it's dependent upon some customers implicitly subsidizing others (e.g. any "unlimited storage" backup solution). It can work a lot of times, but sometimes it doesn't, and at AWS's scale, that could change quickly. It'll upset a lot of customers if they then have to restructure pricing to account for a small subset of customers' use cases being too costly to provide.
You always need the head to subsidize the tail (e.g. free trials). You can’t let the head become unprofitable.
And they can also impose things like bandwidth caps to curb some less-profitable customers' usage. But cloud providers, by design, are charging "per use" and expected to scale "indefinitely"... at least for expected use cases.
Prices should have gone down because they've continued to scale up and go into custom cooling/hardware/backbones/etc.
You're basically saying that you're cutting them slack because their margins have gotten much larger and they haven't decided to raise prices to make them even bigger.
The slack I'm giving is for the complexity in their billing, not in the price itself.
Why should that make prices go down? Prices aren't based on cost. They're based on what the market will bear, i.e., what customers will pay. If you think AWS is too costly, you can go find some other solution. If enough customers start doing that that AWS's financial state suffers, AWS will have to rethink its price and cost structure.
Competition in general tends to drive prices down, yes; but competition was not what the post I was responding to claimed "should" make prices go down. That post was claiming prices should go down simply because AWS is investing more resources in building capital as they scale up. That is not true.
> The fact their prices and profits have remained high for so long indicates lower competition, likely due to high barriers to entry from competitors and high switching costs for consumers.
It could, but it could also indicate that this market simply has not reached equilibrium yet. I don't think it's plausible to claim that AWS has no competition in this area, or that barriers to entry are high; there are a number of huge corporations investing a lot in this market, and plenty of smaller players gaining customers by presenting a simpler interface to an AWS back end (e.g., Heroku). Switching costs might be high, not just for AWS but for any provider in this market, simply because there is so little standardization in how infrastructure is specified and controlled. That would lengthen the time to reach a competitive equilibrium in this market.
It is worth remembering that for new technology markets, it can take a long time for a competitive equilibrium to be established. A good example is the automobile market; in the US, for example, it took many decades for car prices to be driven down to marginal cost and for all of the various market players to search out and capitalize on all the possible competitive efficiencies and strategies. Part of that was also that it took decades for the auto market in the US to become saturated, i.e., for most new car sales to be replacing old cars instead of getting a car to someone who had never owned a car before. I don't think the market for AWS-like services is anywhere close to saturated, which means we should not expect a competitive equilibrium; instead, we should expect exactly what we see, large market players trying to capture as much market share as possible during the growth period, just as major automakers did in the US during the growth period in the mid-20th century. Capturing market share in a growth period is a very different game from squeezing out efficiencies in a market near competitive equilibrium.
My point is that their prices could have gone down without them even feeling any pain because their costs have gone down. They don't deserve slack for keeping prices the same. Keeping prices the same just means their competitors either suck or they sit in an oligopoly together.
Well, there are whole organizations in my company dedicated to installation, procurement, provisioning, ordering(and tracking of said orders), of datacenter resources and hardware.
We are spoiled.
It isn't simple at all! Not only is it complex and fairly daunting, you also need pretty deep knowledge of AWS to fill it out correctly. They eventually introduced an improved calculator that dropped the "simple" branding: https://calculator.aws/
> It's quite a tour de force how Amazon have taken "separation of concerns", applied it to web services and used it to create complex and difficult to understand or predict pricing to print money.
Is applying separation of concerns to web services really that bad? Look, if you're a small company with a simple product, you can just put your stuff on a few EC2 boxes and pay the monthly bill for that. At that size, your infra costs are going to be dwarfed by your other costs of doing business anyway. If you're a big business, you can literally pay people to keep track of this stuff. You've got the extra money, because now you don't employ data center architects, server engineers, etc. AWS is able to "print money" because it brings a lot of value to many businesses.
They don't work. To elaborate, they fail to provide the user with information with enough clarity to allow him to form accurate mental models of what is being used,what it costs, and what is the implication in cost of applying some change.
Heck, at any given moment it's unexplainably hard to tell exactly which AWS services are you actually using, and being billed for.
The only reliable info on cost it the invoice you receive at the end of the month. Even then, the AWS guys felt the need to develop a full blown machine learning service to help customers predict what their next invoice will be.
Did you actually look at the billing page? Here's a screenshot of my billing page:
It meticulously breaks out per-service costs across whatever dimension you want. "EC2-Other" is rolled up in the chart, but it's broken out by each individual item further down in the list.
That doesn’t show what you are being charged for right now, nor does it help you predict what your total costs will be at the end of the month.
Since they don't, if you are literally so concerned with your current costs for this month, you can set a budget and alerts for whatever you want, so that you don't exceed some cost. Also, they show you your current cost for this month, up until the moment, as well as a daily breakdown of your costs.
I didn’t say anything about this at all. It would certainly be a nice feature, and one people have asked for since the very beginning of AWS.
> Since they don't, if you are literally so concerned with your current costs for this month, you can set a budget and alerts for whatever you want, so that you don't exceed some cost.
Again, this is a strawman. This isn’t a response to anything I said.
If you’re trying to understand how much a new deployment on AWS is costing you, and find ways to optimize that, you don’t care about the total budget of the account, or having some auto shutoff. You need detailed, real-time information from Amazon, which I’ve never seen in the console.
It’s been several years since I’ve had to be digging around in the billing console, so I have no stake in this discussion and things might have changed. Based on other comments being made around here, I really doubt anything significant has changed.
I was just trying to clarify a comment you completely misunderstood.
No one who is working on this cares about real-time by the second data - the daily cost is enough.
> Again, this is a strawman. This isn’t a response to anything I said.
OK, what are you trying to say? If my response above did not answer the question, can you give me a one sentence summary of the point?
>> I was just trying to clarify a comment you completely misunderstood.
Alternatively,
“The billing UI and billing infrastructure is a major weakness of AWS when it comes to helping people understand what their usage will cost.” You can clearly see this is true by the large number of people who complain about it.
- There's almost no demand for it from the majority of their customers.
- Billing code is a big, scary, tangled mess and also happens to be the one area where you really can't afford to introduce bugs and make mistakes, so meaningful change is already exceptionally difficult.
Amazon, in my experience, has issued organizations a one-time credit in cases of surprise bills.
The bigger crux of the issue, is Amazon has democratized CapEx into OpEx, and it's too easy to be ignorant of the details of infrastructure planning, which is rolled into AWS pricing.
Before AWS, IT teams had to plan for and estimate costs prior to procuring hardware. This is no longer needed with ondemand, programmatic infrastructure.
Engineers, who (reductively) focus on solving business problems with code, aren't necessarily thinking about the costs of deploying their solutions, the way Ops and IT would be.
I already got a surprise, because in one place it says t2.micro and t3.micro instances are free for twelve months.
But, if the instance runs Windows, or the zone is this or that, or something else I don't know, the only free tier instance is actually the t2.micro.
And then there's the Unlimited option, and CPU credits, where if your CPU use goes over 25% for some time, it will either: Be throttled so slow as to be useless, or, not being free and you still have to pay for each hour of use.
And that's just the beginning!
What I need is clear and upfront documentation about pricing, before the bill comes.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
A lot of people use Terraform (my shop does) and I've found a good terraform setup way better than CF templates. You can do tags in Terraform too. You should be forced to use their already bad tooling just to get correct cost estimates.
Blame Hashicorp if they don’t have a similar feature.
Also if you are at any type of scale, your IAAC choice is the least of your problems.
Have you actually done a realistic project plan and estimated how much it would cost to migrate infrastructure from one provider to another once you reach any scale - including regression testing, auditing, training, etc? Do you really just use your cloud provider to host a bunch of VMs? Even then it’s not that easy at scale.
Yes I've dealt with this issue (to be more precise multi-cloud rather than migration, also used other TF providers in tandem like the Kubernetes one), and yes your TF resources are provider-specific, but being able to handle it all with the same tool instead of having to deal with an awful vendor-specific tool (to be charitable, because the reality is having to deal with mutiple ones and having them inter-operate through bash glue) vastly helps in reducing and controlling lock-in, and I would argue it's the baseline step you have to take if you don't want to risk your whole business on a single cloud provider not going to the gutter either because they've grown too big for their own good or because they're being squashed.
CloudFormation means jumping into the pit of lock-in and you can only climb back out by digging your fingers into the dirt wall, Terraform means you have climbing gear to rappel down and if you have to, get back up without as much hardship. Sure you have to put the gear on but your descent is controlled and then you can climb back up at your own leisure.
If I get to make the call and I care one bit about the business beyond next quarter, I would always have a clear way out to heavily reduce risks to a situation that we have seen play out many times before with the Oracles and IBMs of the world.
You can’t imagine the red tape moving from Workday and their homespun EMR (Healthcare) written 20 years ago using Powerbuilder.
I get that there are times to dive all the way in, but there is still a part of me that says "Is this what the Internet is now?"
Terraform in no way provides you an abstraction layer over the different cloud providers.
A unified abstraction layer would certainly work for a subset of cloud offerings. Think the more basic stuff like EC2, VPCs, etc. Platform specific extensions could be handled as optional arguments to the unified objects.
That said, even though you need custom Terraform definitions for each cloud vendor, using one IaC tool still beats having to use a cloud-specific one for each of your deployments. The parent's point stands.
Learning the different IaC tools for each provider = Learning Python, Javascript, and PHP.
Also, Terraform does state management and can be used to package deployments of apps to different platforms. Our IaC pipeline uses Terraform to deploy resources to AWS and several other cloud toolsets, all in one language.
Using Terraform vs CF is easy compared to using AWS vs Azure.
Given that AWS supports terraform, it's really on them to provide a calculator for it as well. We're large enough that we just spent an obscene amount of money on employee time and effort to track our billing (lots of automated tagging), but until we did that it wasn't uncommon for a couple hundred thousand dollars of unnecessary charges to hit our bill every month.
Second, the "in-between" costslike bandwidth from service to service, iops on your disk, ... They mostly don't exist on prem, you either have enough of it and it works or you don't and it doesn't work and you need to go buy a bigger pipe. They're one of the biggest source of surprise cost on AWS.
Eg take a basic data storing application, the cpu and storage space are easy to estimate and are the same as on prem, but suddenly you also need to estimate your iops, and you realize you have no idea about it. Do you have an on-prem database running somewhere ? How many iops does it need ? Unless you've had to figure it out, you have no idea and will never know on prem.
I’m often fielding questions over on r/aws about how to do X on AWS where X is something simple. I then ask are they trying to solve problem X or are they trying to learn AWS. If it is the former, just use LightSail. Outside of serverless applications, I wouldn’t want the complications of full fledge AWS for a small project and I get paid to know the ins and outs of it.
It could be there are such tools. But they are impossible to find, setup or understand for someone who is not already an AWS expert. I recently got a $500 bill from AWS. It was impossible for me to find out which S3 buckets (or even specific files) caused these costs. I looked at this for half an hour and have used AWS before. They just don't tell you how to obtain this information. Or maybe they do, and it is so well hidden that it might as well not exist. This was a horrible user experience. As a customer, I want to know what I'm being charged for. The fact that AWS make it so difficult make me never want to use them again.
EDIT: sorry, I forgot to include the screenshot link - https://imgur.com/a/tVFoocr
https://aws.amazon.com/premiumsupport/knowledge-center/s3-fi...
Why do people think that they can give Amazon a credit card number and expect everything to be easy enough for them and suitable for a large enterprise? AWS offers something for beginners - Lightsail. When that outgrows their needs, they can peer their behind the scenes LightSail VPC to a full fledge VPC.
The biggest mistake I see is that people jump into AWS whole hog without doing the research or if they are large enough, hiring someone who knows what they are doing.
https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2...
Just like AWS is not understandable to someone new, I feel the modern front end ecosystem is royal cluster but yet and still thousands of people understand it after a 12 week boot camp.
It's not my intention to victim-blame here, yet do bear in mind that there are many, many ways to use the services and to expose resources to the bottomless appetite of the internet. Maybe Digital Ocean or Linode or the like would be a better option for starting out.
Yes, I use it extensively.
> Is applying separation of concerns to web services really that bad? Look, if you're a small company with a simple product, you can just put your stuff on a few EC2 boxes and pay the monthly bill for that. At that size, your infra costs are going to be dwarfed by your other costs of doing business anyway. If you're a big business, you can literally pay people to keep track of this stuff. You've got the extra money, because now you don't employ data center architects, server engineers, etc. AWS is able to "print money" because it brings a lot of value to many businesses.
It's not bad per se. The real problem I often see is that many people within the typical org aren't aware of how these costs can accumulate, especially when you have multiple DTAP environments, microservices, and so on.
Another problem is the days of "put your stuff on a few EC2 boxes" seem to have been overtaken by the cloud native, microservices, k8s trends of the last few years. Personally I 100% agree this is the way to go for many (if not most) people -- and for this purpose Digital Ocean is just as good -- but what I see more often is people diving straight into the deep end, because "Is applying separation of concerns to web services really that bad?"
I also think even big businesses underestimate the total costs of these all-in cloud infrastructures. You indeed need to pay people to keep track of this stuff -- multiple people. Your cloud costs + your devops/platform personnel costs can get to be a significant % of your total IT costs. Is it really necessary?
Just because your business has a big market cap or significant revenue from digital doesn't necessarily mean it needs any of that stuff. Chances are you can run it all on the same couple of EC2 boxes you ran it on when you were still a small company.
Maybe a good analogy is to think of AWS as C. It's flexible and powerful, but it's easy to shoot yourself in the foot. That would explain why a lot of these "> magical-tool deploy" services have emerged, promising to abstract away AWS.
I am not sure where you get the notion that any serious user goes with the "put your stuff on a few EC2 boxes" approach. Every single AWS project I worked on had a cost calculation phase when we investigated what combination of services could be the most cost-efficient for a certain problem.
>> cloud-native, microservices, k8s trends
It is kind of funny that you try to put on one end AWS the company that made cloud computing popular (and microservices with it) and the biggest player in the segment and "cloud-native, microservices, k8s" on the other. Trends come and go, AWS is here for the long run.
With something like digitalocean my costs are fixed. I might have misjudged how much I need but can scale up easily or down (with effort).
Will Amazon offer fixed pricing? Does Amazom offer fixed pricing?
For a small/startup, the ability to execute quickly without sinking capital into a product that will likely pivot anyway is a superpower.
When you get bigger, stuff gets dicey. Paying a premium for a "X as a Service" made sense when you had no people, but at some point the cost structure may not make sense or your needs are at odds with how the service delivery works.
When you work for a big company or government agency, things change again -- you don't hire a database guy for a project, you have a project team that builds stuff, and an ops team that exerts the minimum viable effort to keep your stuff running or an outsourced ops team that does the same, but worse. At that point, you start looking at how much it costs to pay AWS to provide and SLA vs hiring more competent (ie. more $$$) ops people to keep it running.
The long and short of it is that AWS is in a position to rake in cash up and down the spectrum -- all they need to do is be competent, which they are very good at.
One can mention that there is a LAMP stack behind WordPress without knocking too many unconscious.
Amazon's genius has been to sell the various permutations of the acronym "LAMP" as services.
Although I guess Lightsail would be a lot more affordable.
I faced a similar issue at the company I work for (where a PM had activated an instance for months and racked up a $9k bill). I've been spending my coronavirus time hacking together a service that hopefully makes billing on AWS more clear. It does cost optimization and anomaly detection right now, and I plan on adding more features as I start to see more use cases.
Check it out if you'd like, and let me know what you think!
www.usage.ai
There's a level of complexity and nuance in AWS that casual users or those on the outside do not quite comprehend. And honestly it takes quite of bit of experience and interaction to fully understand some of these things. It's not that you're getting charged "twice" for S3, it's that there are separately line items in the CUR for S3 data transfer, S3 storage (and S3 API, S3 inventory, etc.). It's confusing to a layperson but this level of detail is actually quite powerful. You can tune your application to use just what's needed.
I look at the whole stack and aside from IAM and EC2, and ECS I think you could get by without a lot of those products.
Yes you can probably launch your own cache fleet using a bunch of ec2 boxes - but you don't get all the low level hooks Elasticache gets. You don't get the pricing discounts either (I remember cross zone replication or something being distinctly free for ec2 but you pay for network. Same with dynamo - access from all zones cost the same pretty much IIRC).
If you are going to make the dive, it is worth spending the time. I would do a small overview bootcamp somewhere so that you know what's available and then start digging in for each new service you take up.