MVPs and $100k AWS Bills: Reflections on our launch
octopus.com
octopus.com
> We also didn’t have our pricing quite right. We initially launched Octopus Cloud at $10/month, with a different pricing model from the one we currently use. Unfortunately, this was one of the most painful lessons we learned because the deficit between what we were charging and spending was magnified by the sheer number of people using Octopus Cloud; continued growth would further amplify the problem.
"Every person who fills in a web form gets to burn a $100 bill of ours" is the real disaster here.
I know it's trendy to think that software isn't like regular business, and you can lose money but make it up on volume, but going to a cloud vendor gives you a real cost of goods sold, and that needs to be accounted for.
(also, the irony of a company whose product appears to be about cloud deployment having a disaster by deploying to the cloud should not be lost on us)
Another mistake, it was pointed out by Kevin Hale, iirc, this year at startupschool, that founders make is they think their pricing model (or their UX) is a key differentiator, when in most cases it isn't.
Yes, I have seen that happen a few times in SuS (I'm a 2018 and 2019 batch alumni). Founders are obsessive about their pricing and as a result they usually undercharge.
My usual advice to pre-market-fit stage founders is that you don't know yet what the right price is, so choose $random_price and move on, you can always adjust later. And for post-market-fit stage: raise your prices.
Of course, there is no single 'right' answer on pricing. It depends on many factors, but for B2B it usually holds true that pricing isn't that much of a differentiator.
* Lets you easily explore a new business offering
* Without investing a large amount of engineering time
As soon as they realized they had significant demand and horrible unit economics, they should have revised their cost structure/app architecture.
Which is exactly what this post is about.
AWS has something like profit 25% margin. Therefore assuming that AWS is within 15% of doing a half-decent job at controlling costs, that's still max 40% profit margin if you were running bare-metal and beating AWS at their own game to the tune of 15%. So if you're selling stuff for 10/month, your AWS bill should be 17/month just to break even after all the awesome engineering you've done with the VC money -- at least that's how I see it.
Now, if they are doing incredibly stupid stuff, to the tune of 90% wasted ec2 cpu cycles/db requests, then maybe they've priced it right -- who knows.
Would love to hear some insights on where I might be wrong/misjudging here.
AWS has (taking your number as correct) 25% margin at immense scale. Hardware is cheap at this scale. unit pricing for power and space is cheap at this scale. But labor cost is enormous at this scale. Even factoring in leverage of automation, at this scale there is just enormous bureaucracy. One guy or just one team just can't "do something". Anything worth doing will have enormous repercussions across many areas and needs to go up and down the chain. Stuff still gets done, because impact is large at this scale.
> 90% wasted ec2 cpu cycles/db requests,
So then, my point is, if you are just using EC2 and egress, AWS is very expensive. You don't need all the big scale engineering they have, to launch an MVP. Of course, you need to be able to hire the type of people that can manage in-house compute, and that may be hard to find these days, at any price. But taking labor shortage out of the calculation, it's vastly better to DIY.
If you are using PaaS features, that's the sweet spot where you get tremendous value out of AWS.
> profit 25% margin. Therefore assuming that AWS is within 15% of doing a half-decent job at controlling costs, that's still max 40% profit margin.
Getting back to this, even if you dispute all of what I've said, your basic math is wrong. 15% short on 25% is 29.4%. (25/.85, not 25+15).
It's got to show up in the books somewhere.
I see the theory but $100k/mo in AWS fees is madness. They could probably knock a zero off just by deploying in containers.
Is that really irony though?
I would think irony here would be them writing a blogpost about their troubles with their cloud deployments.
https://hardware.slashdot.org/story/11/06/21/045228/dropbox-...
https://www.theregister.co.uk/2015/01/17/scary_code_of_the_w...
Edit: I know most database applications like Postgres can handle multiple databases. Does RDS let you have more than 1 database per instance?
Especially since the DB is SQL Server; the license costs are death for that. An open source DB would save you a lot there if you have a real need for a DB instance per customer architecture.
No, they mention in the article that they were using 30 databases per DB instance. Not sure why they used 30 when the RDS FAQ mentions[1] that for SQL Server you can have up to 100 databases per instance.
However if they would have used another database engine such as MySQL or PostgreSQL, they would not have had any limits to DB's/Schama's per instance and could have had an order of magnitude of cost savings. Using something like AWS Schema Conversion Tool[2] makes it easy to convert between different DB engines, and at a spend of $100K /month they could have easily afforded to pay a consultant to do the migration for them.
[1]https://aws.amazon.com/rds/faqs/ [2]https://docs.aws.amazon.com/SchemaConversionTool/latest/user...
I have used octopus and I could have told them that they can't lift-and-shift. The amount of requests the app makes is ridiculous, the way they wrote their database is appalling (actually crazily dumb https://octopus.com/blog/sql-as-document-store). Everything about it would mean they can't scale in the cloud without wasting a ton of money.
"Cloud stuff can be really expensive" - yes if you write dog shit, putting that in the cloud will be expensive.
"For a many-to-many, we do something a bit naughty: we store the values with | characters between each value:"
Not naughty, stupid.
I haven’t read the post... maybe I’m missing something.
It's non-1NF, though they sometimes (based on expected cardinality) normalize that aspect via a trigger, using the application-accessed table as, in effect, something of a write-through denormalized materialized view. It's not an entirely unreasonablev way to use an RDBMS in and of itself.
Lifting and shifting is hard. It's harder when you do it in a careless way.
Why didn't they just make 1 table with 1 column and put a JSON object in it with all customer data? This would fit their design objective "Avoid the need for joins as much as possible"! It's brilliant! We'll call it: noSQL, because we don't even have to write that pesky SQL!
Can't wait for the next blog post where they discover another naughty trick called primary keys to increase their NoSQL performance even more!
I can't do it clientside. Nature of the problem.
I'm writing it in Rust, so it's relatively fast. But throwing thousands of people at it will still require extreme scale. I'm going to use K8S horizontal autoscaling. This will be obscenely expensive.
I'm not sure I can afford this. But I think a month of successful demonstration will get me funded (or bought out). So in theory my costs could wind up getting paid. Maybe.
Do I risk launching this and potentially having $100k multiples of AWS bills? I can't afford that. I don't want to deplete my life savings or potentially go bankrupt.
Could I put it on a corporate card as an LLC and do my best to make it work? Then fold without harm or fowl if it doesn't? Is that unethical? Could I get sued?
All that said, I'm almost certain that a successful demo at scale with the Internet thrown at it will get me money. It's a really cool and novel product. People will be talking about it.
What should I do? Advice will be very appreciated.
you don't need to go from 0 - 100k a month in a single month, octopus could because they have backing.
If you need funding, then you have to build a compelling case sans demo. It's not unheard of.
If you have no clue how to monetize it, why would anyone fund it or buy it? People fund or buy businesses, but without a plan for monetization, you don't have a business.
TL;DR Pay someone who knows how to work with baremetal servers, convert your $100k/mo AWS bill to $20k/mo infrastructure geek bill + < $10k/mo infrastructure bill.
It will give you a longer runway.
Nobody is going to give a line of credit to a brand new company (LLC or not) without a personal guarantee from the owner.
> All that said, I'm almost certain that a successful demo at scale with the Internet thrown at it will get me money.
I would not rack up $100k of personal debt to get to a demo.
But maybe don't listen to me, I'm a 40 year old programmer with a few failed attempts at bootstrapping behind me. Quite possibly at least one of those died due to being capital starved :)
>Could I put it on a corporate card as an LLC and do my best to make it work? Then fold without harm or fowl if it doesn't? Is that unethical?
Any bank worth their salt won't simply hand over a huge line of credit for an idea. Banks are risk averse, that's why VC exists. But hey, if you are honest with your plan and they give you money, this is the route.
I'm hoping for a quick viral demo, press coverage, and then either more funding or a quick exit. Enough funding to handle the intital traffic will absolutely accomplish what I want.
I've already written the app, so I can show it off.
This is a great idea! Thanks so much! :)
I am not saying that's not true - but that isn't enough to make it a product (or being bought out). Some of the most popular products out there aren't hard to do. The complicated part is how to scale them to be as fast with that many users and don't lose money. Twitter? It could be done in a few months (I am not saying it is a weekend project), but that doesn't mean you have Twitter. The hard part of Twitter is not tweet, publish, have followers, is doing in almost realm time with that amount of data. You could say almost the same for YouTube (Upload and play video? Easy. Do it for millions of video at the same time... oops!).
Try to optimize your backend, so it doesn't cost much money. If not, you have just something that does something nice but it doens't do it well. Many people can do that. The secret of success is doing great things without consume many resources.
Do some maths.
Get one (1) small EC2 instance on the free tier. Deploy the test version there. Keep adding test clients until the thing is maxed out (presumably on CPU for your application?). That gives you a ceiling for "peak usage". You may find this is surprisingly high.
Come up with a ballpark figure for how many client requests a normal user will make in a day.
You can now start producing cost numbers for using various solutions, from AWS to dedicated hosting. If you don't have deep pockets, do not choose an option which can incur unlimited bills.
Before you can even think of selling it to someone, you need to have a compelling revenue story. Not actual revenue, but a convincing path to revenue. These days that's usually advertising. Can you bung some ads in there from Google to cover startup costs?
Which would you rather have, the system page you in the middle of the night to add more instances manually, or the system do it on your behalf without regard to cost?
Or limit the number of beta users there can be.
Or work some freelance jobs for a few months and save up the cash and during that time pitch to investors. As you progressively have more cash you can hire an engineer to part-time hack on cool demos or to give you the opportunity to increase the number of beta users.
First, the deployment was on AWS (EC2 and RDS among other services) which was probably already co-mingled with other AWS customers.
Second, when you are selling a SAAS offering, the assumption is already there that you'll build security and multi-tenancy into the app and the underlying infrastructure shouldn't matter.
And 100K/month for a MVP SAAS offering. I'm speechless. I just, I can't even understand how this burn rate passed muster with the leadership.
But Kudos to the team for owning up to it and sharing. Lots of lessons learned here.
Cloud CI platforms typically solve this by timesharing build agents, but that takes engineering effort, which is a waste of effort if there's not a lot of demand in the first place. That's why we started with the MVP.
Unfortunately, I’ve met way too many old school netops guys who got one AWS certification and all they knew was how to log into the web console and duplicate their colo infrastructure, charge their client a lot of money, and call it a day.
That being said, I’ve used a lot of CI/CD tools and OctopusDeploy is by far my favorite.
Even though my company is very much a “pay someone to do the undifferentiated heavy lifting” type of company, OctopusDeploy is so easy to use and maintain, I am not sure we would migrate.
Going to the cloud lets you scale beyond your own data centre
Going to the cloud can be cheaper if you use the elasticity and make good use of PaaS services (servless,paas, etc)
"To ensure there was no way for one user’s data to mingle with another, each cloud instance had its own dedicated VM, database, and a large number of security configurations to prevent any funny business.". AWS has a myriad of security and isolation tools to do that in significantly better and cost efficient ways.
"Octopus Cloud customers could start a free 30-day trial, which meant that those hundreds of trial signups per month, each of which cost us $100 to host, quickly added up." So you're essentially allowing anyone to press a "Submit" button to spend $100 for you. This sort of below-cost methods to gain marketshare works for Amazon, Google, Microsoft and son because they have the deep pockets but it's a questionable strategy for smaller companies.
At that level, at the very least, it would make sense to run some amount of physical infrastructure.
$cost_of_a_thing < $charges_of_customer.
$cost_of_a_thing + $additional_expenses_allocated_to_ thing could be > $charges_of_customer
Described business model:
$cost_of_a_thing > $charges_of_customer.
hence $cost_of_a_thing + $other_expenses_allocated_to_thing is > $charges_of_customer
First one can be profitable via scale. Second cannot.
What we didn't know if whether there'd be enough customers for us to care about the loss on those $10 tiers. E.g., if only 100 customers signed up a month, the costs would have been around $10K/month - enough to write off as a marketing expense. Once it got to $100K it became something we needed to fix.
Err.. That's not a correct way of looking at a long tail. CGS on a long tail must still be lower than the unit charge, otherwise you are paying $1.00 for $0.90 which gives you an incorrect signal about the demand.
Say you make apple juice. And each juice uses 4 apples and only apples. 4 apples cost you $3. Setting a price of juice to $1.00 does not actually tell you if there's a demand for the juice because it is below the CGS, which you cannot make up in volume.
The long tail test is to set the juice price at (CGS + x) where x is some number and see if you get the demand even if x is less than the other costs attributed to juice ( rent/electric/juicer/salaries). If you do, then you can make up your loses in volume because the higher the volume you are going to have the smaller process cost attributed to every unit will be so there's going to be a number at which CGS + y < CGS + x where y is the process cost attributed to a unit.
Or find a location (rent$), get fiber($$) to that location($$), find hardware($$), install said hardware. Now you’re back to where you were with AWS.
For larger already established businesses on prem is probably a setup you should explore but on day 0 of a new business, cloud will probably have a large cost and flexibility advantage as long as you smartly set it up.
Total first year cost on this setup: $8,800
Total three year cost on this setup: $12,400
2x m5d.4xlarge EC2 instances (this isn't counting ingress/egress rates, or really anything else) First year cost: $15,621 .
Total three year cost for the EC2 instance: $46,863.36
For the cost of using EC2 you could hire an extra person to help grow your company.
Yes, you can get discounts in AWS for agreeing to a three year term. Yes AWS offers you flexibility to walk away at any time.
If you have a base line load (which most companies do) you should really look at colocating your owned equipment. You also get the bonus of being out of the watchful eye of Amazon's profitability team looking to see if they can copy what you're doing.
I think, at least for some business types, colocation will usually be more expensive because you have to overprovision or risk losing customers.
You _will_ be hiring an extra person to manage these machines. That's the beauty of The Cloud (tm). Need more compute? You got it. Provisioned too much? Get rid of it. If you're a startup, you don't have time nor money to be managing your own infra.
Not to mention the ability to deploy to _any_ region within minutes. Now you're talking $8,800 / region + someone to maintain other regions.
> ingress/egress rates
unless you're also using S3 or CloudFront, this doesn't really mean anything. Put a free CDN in-front of S3 or your instances if you're worried about load or cost of IO.
> You also get the bonus of being out of the watchful eye of Amazon's profitability team looking to see if they can copy what you're doing
Ok now you're reaching.
There are colocation providers all over the world. Psychz.net is a great example. My assumption is a single box per region. There is no redundancy in your ec2 systems.
You should google the Amazon profitability team if you don't believe me.
Tangentially related: https://www.datacenterdynamics.com/news/ovh-ceo-unlike-amazo...
https://sellercentral.amazon.com/forums/t/amazon-literally-s...
https://fortune.com/2016/04/20/amazon-copies-merchants/
https://www.theverge.com/tldr/2019/9/19/20874818/amazon-allb...
But what are you getting 2x m5d.4xlarge instances from the get go? Start with the smallest instance size your app will run on and then add them on one at a time? Need to run 10 servers in the day but only 2 at night? Then configure autoscaling to do that. What happens if you need to expand out into another part of the world? Okay, now you just copy your setup to another Region instead of trying to find another provider in another country.
What if your service becomes super popular (You got on the front page of HackerNews), with autoscaling, it just grows automatically. No scrambling to get more servers, it just scales.
Don't listen to the "Woe is me!" stories about using the Cloud and feel sorry for these companies. They designed poorly, wrote crappy quick code and paid a price for it.
Except S3. Unlimited objects, and unlimited Put and Get requests.
Do you know what exactly a 'prefix' means in this context? The first N characters? What is N?
No this will not protect you. Understanding the proper way to configure IAM roles with principle of least privilege will.
The author mentions that The issue here is Octopus is like a CI server - customers can run arbitrary code in the instance (not a problem most SaaS apps have). So a VM per customer was almost the bare minimum.
If you're using the same IAM role for your EC2 instances, and you're allowing your customers to run arbitrary code, well now at the very least they have access to everyone's databases.
Even accounting for hindsight being 20/20...every customer gets their own VM is a rather ambitious approach to scaling.
Would love to know what made them shift to Azure. Especially while embracing linux at same time.
My questions is, is it still aimed at .NET? I know it leans on Nuget as the package repo. But have they broadened to embrace deployment of other tech stacks?
Is Octopus considered best in class for what it does? Who are the primary competitors? Does Azure DevOps spell the end for Octopus?
We decided to build an MVP based on our best estimates and test the market that way. The goal was to launch something in 6 months and test if the demand was there; and if it wasn’t, we’d only wasted 6 months. We chose to optimize for getting to market quickly rather than worrying about how much it would cost.
Contrast it to what the Lean Startup principles say:
The Lean Startup methodology has as a premise that every startup is a grand experiment that attempts to answer a question. The question is not "Can this product be built?" Instead, the questions are "Should this product be built?" and "Can we build a sustainable business around this set of products and services?"
So instead of doing what a lean startup would do, they did the opposite. Let's see what they could have done instead...
"How many customers will actually use it?"
- Lean startup: create a signup page with a mock up or landing page and a few buttons that work and every potential behind a "coming soon" screen - Their approach: actually build the whole product and then launch it and see who's interested
"What is it all going to cost?" - Lean startup: As little as possible to discover the insights we need to iterate and evolve the product - Their approach: well we already paid the salary for our engineers and we've given them six months, so what's half the salary+benefits of X number of engineers?
"What should we charge for it? Is it going to cover the infrastructure costs?" - Lean startup: experiment on costing and pricing as you go along - Their approach: give away the product for a 30-day trial at a high cost to ourselves! Unbelievably expensive customer acquisition costs.
Reference: http://theleanstartup.com/principles
And now I guess I've truly learned why product managers exist. Someone let this crazy experiment run for six months. You're saying in six months you couldn't spend one month finding the simplest/cheapest ways to test these various hypothesis?
$100k customer acquisition spend is insane.
The whole post is just a lesson in contradicting what an MVP minimal viable product is.
Even when we were building v1, the team knew it wasn’t the ideal architecture. Before v1 even launched, there were plenty of conversations in our Slack about whether we should port Octopus to Linux and run it on Kubernetes, or see if we could run it on Windows within Kubernetes or use Nomad by Hashicorp?
Lol, engineers always want something fun to work on and sometimes it turns out to be a great idea. But they porting software for an MVP? Why not evolve the new product so that it generates more revenue or reduce the costs in some other way instead of talking about porting and rewriting?
If you're at an established company that has money to burn, go ahead, read the whole blog series. If you're trying to create a new startup, whether it's a product or service, avoid this article or read it as a warning. When you've spent as much as a decent engineer costs in one month...something's gone off the rails.