Principles for Building New SaaS Products on AWS
trek10.com
trek10.com
#4 Operate as if you may be bought or cloned by Amazon at any time.
Once you have any type of meaningful infrastructure you’re already not going to migrate without major pain between data migrations, networking, security and compliance audits, retraining, etc.
And if you are only using one of the major cloud providers as a glorified colo, you have the worse of both worlds - you’re paying more for infrastructure and your TCO is no lower because you’re still babysitting infrastructure.
Even almighty Amazon took a decade to move off of Oracle. Concentrating on avoiding being tied to Oracle early on instead of building a business would have been silly.
Trying to avoid “vendor lock in” when you’re still trying to grow your business, when your focus should be trying to acquire customers, and raise revenue and profit is the ultimate premature optimization.
it depends on your core business. Don't be "vendor locked-in" for any core business competencies. But do get "vendor locked-in" for non-core competencies. If you're core competency is merely just white-labelling another business's offering, then obviously it's a bad practise. But for things that are easily commoditized - like CRM and payroll, why not be "vendor locked-in", and save the cost of development, and enjoy the lower scaling cost?
If the price per seat is 5.00 dollar per person and next month it could be 5000 because they require you to pay surge pricing getting locked on could kill your business quickly.
Do any cloud providers have surge pricing? (I've not read to many SLAs myself.)
AWS has never raised the price for a service since it existed.
I can still launch old instance types which are no longer listed. Same with SimpleDB still works.
I also haven't seen the reverse, the multi-cloud people who later on drop a cloud provider because of some reason, going "omg, thank goodness we didn't let ourselves be locked in to one, this saved so much time/money/effort".
It basically a fixed cost imposed by the cloud providers.
I.e. when the economy contract, and yet AWS/Azure profits are growing, this is the pain of vendor locking.
How many companies that were on such thin margins between surviving and going out of business that cutting infrastructure spending by 10 or 20% was going to make a difference in them being able to survive?
Which is, vendor locking did not show itself, because the economy was good.
Any large company depends on literally dozens of vendors that they have tied their business process into where it would be a major pain to switch.
I know in the health care industry, a lot of health systems are so tightly tied into their EMR/EHR, leaving a something like AWS would be a breeze.
Smaller companies often run leaner. Except maybe for big airlines that spent their surpluses on stock buybacks and executive pay increases.
There was something in the last month where somebody closed their company because such a mistake took them through many months worth of bootstrapping budget.
I don't know if that counts as regretting the lock-in, but I guess they regretted something about their cloud provider.
Did that for a web startup. Tens of services, 200 instances.
Estimated 1-2 weeks to move to another cloud provider (Google/Azure) and be back online for customers.
100-200 VMs - especially if they are all based on the same few image is easy.
I am considering a disaster recovery scenario. Like AWS banned the account or all the resources were deleted. There is nothing to do but to start over from scratch, every single employee is onboard to help and get their bits of services back working. Audits is not a rebuild step, it's something to arrange much later.
In general, if you build apps with open source runtimes, use open source dbs, and avoid the proprietary data services cloud providers really stick you to, you can move around pretty easily. And you still reap most of the benefits.
The real problem with the special AWS services is you end up having to hire AWS ops people or expensive consultants to architect around them and run them for you. So it's proprietary AND eating up salaries.
But stuff like SQS and (to a lesser extent) Lambda is really hard to replace because it's thoroughly baked into an application architecture.
As an example, we (fly.io) have a tool that'll hoist a Fargate app into our infrastructure and let you run it all over the world. We even have people tunneling back into their VPCs to access other AWS services. But that only works because we're both somewhat standard, Fargate takes a Docker image and runs it, Fly takes a Docker image and runs it, the app inside doesn't care about either of us.
Have you actually costed out how much a large migration would take?
As far as using Lambda for an API, you can literally add three or four lines of code and use “proxy integration” to deploy your entire Node/Express, Python/Flask/Django, C#/WebAPI app on lambda and without changing any code, deploy it anywhere else as you would any other API.
Here is a Node Express example.
And, most of what you're talking about doesn't really affect margins for a SaaS. Every AWS service hits margins, one time migration costs and even R&D time to build products does not. The marginal cost of using AWS underneath interesting features is very high.
Every dollar you spend on R&D or migrations you have to consider whether that dollar could be better invested somewhere else and whether it adds business value to have the expertise in house or to outsource it.
Dropbox and Backblaze for instance decided that storage was a core competency. Dropbox moved away from AWS and backblaze knew from day one not to get on it.
On the other hand Netflix went the other way and is now AWS’s largest customer.
Here is the sum total of how much “work” you have to do.
const awsServerlessExpress = require('aws-serverless-express')
const app =require('./app')
const server = awsServerlessExpress.createServer(app)
exports.handler = (event, context) => { awsServerlessExpress.proxy(server, event, context) }
How much time do you spend babysitting infrastructure and how much money is it making your company or saving your company? How many of your customers care about your valiant efforts at “avoiding lock in”?If you build the app around e.g. Lambda and DynamoDB, in contrast, you're going to have a harder time both because you need to restructure the code but also verify that what you switched to doesn't have key differences in how things operate.
Dynamo is tough. At the same time, it's really really good.
It's impossible to use a cloud provider without using any of their managed services; if you are using EC2 or a similar IaaS as if it were just a basic VPS, you'd probably be better off (cost for the use) with a basic VPS, but you'd also probably be better off with Amazon LightSail, which more closely approximates a basic VPS service.
What more is EC2 than a virtual machine? Sure you have it tied into IAM that's tied into firewalls and whatnot, but that's doesn't make it radically different from a VPS.
A cheap VPS provider or a colo do not give you storage and networking. There is no cheap and no open source solutions to do any of that.
We build an open source solution[1] to deploy autonomous Kubernetes clusters into on-prem or air-gapped environments but it's also useful for limiting cloud lock-in (even has its own "IAM" built in). Of course, you have to also limit your use of proprietary services (which definitely has its trade-offs) but might be worth poking around if you believe reducing lock-in is worth it.
(Not baiting just curious)
Aws passed 1mm customers (businesses in 2015-2016)
If this is the case do we not likely run more risk having our landlord, best friend, VC partner, banker, vendor, first X hire... also steal our business/model and it be more likely than AWS?
I personally take using Cloud9 to the absolute extreme(https://www.trek10.com/blog/i-buy-a-new-work-machine-everyda...), having my Cloud9 env setup scripted and creating a new one every day/project. I don’t really recommend that approach for most folks. Anecdotally, it has paid off well when I left a Mac on a train and I was able to walk in an apple store grab a new one and lost minimal productivity for the day.
However, the flip side of all this is I regularly work with a lot of IT people that have underpowered machines, flaky / poor internet, crazy restrictions on their work machines that cause all sorts of problems with CLI / program installation, etc. I’ve found Cloud9 to be super liberating for those folks particularly with the parity of Cloud9 to AWS Lambda runtimes.
The 95% is still local, but makes remote more ok when ci + jupyter + quick vim isn't enough.
I am a big fan, have been since before Amazon bought it. I’m shocked more vendors don’t have something similar.
I write about AWS products and it's quite nice to get an IDE preinstalled with AWS CLI tools with the push of a button.
For JS it's mostly an okay-ish IDE, not as good as VSCode, but okay.
For things like Rust or Reason it sucks quite much.
I wouldn’t be opposed to a cloud-based primary IDE (as long as it functioned offline), but I don’t think we’re there quite yet.
So it’s not my primary IDE, but it’s a solid backup.
Also, not everyone has fast/reliable internet all the time :)
The majority of HN readership I’d encourage to continue using their own tooling, you’ve got fast internet, unrestricted access and powerful equipment.
That all said, I default to Cloud9 these days just so I can bounce around machines and have a consistent dev environment when I need it. A lot of my daily job is meeting teams where they are and helping them be productive fast as possible so I need to stay semi-fluent in most operating systems.
(Compared to Cloud9, I greatly prefer Workspaces, but still use Cloud9 on occasion for a few niche use cases)
I treat Cloud9 like any other ephemeral editor process. Need a new editor window? Cloud9 project. Done for the day? Commit everything I care about. Tear it down.
That said, I frequently spin up Windows workspaces to test software or workflows if I’m writing a guide or content.
- Build your application to suit many customers or one. Large customers love to have their instance run in a dedicated account.
- Open source your operations. Large customers love to see logs/operational activity from their environments (e.g. cc: cloudtrail/config/cloudwatch logs to customer, this assumes dedicated account)
- Open source your security. Be prepared to ship guardduty, config rules, etc to your customer (this assumes dedicated account).
Also, since we're focusing on AWS here
- Build your application to support hybrid cloud customers. Expose it through private link, VPC connections, transit gw, firehose, api gateways, VPC lambdas, whatever is appropriate for your architecture.
- Leverage IAM as much as possible for authentication/authorization. Not AWS-specific but implement SSO and assume customers will require numerous instances of your service when developing your user account model.
- Leverage KMS as much as possible to protect data at rest, and support customer CMKs.
There are more but I'll stop here.
The latter just means "it's in [version controlled] code". This has a variety of use cases, and it might mean you end up in a quagmire of complexity. It's become a cargo cult thing where I've seen people adopt horribly complex, fragile solutions over practical ones "because IaC".
The former is a principle that basically has no downside, and only improves operational integrity. Even if you're literally deploying everything by clicking in the Console, it's still massively more reliable, repeatable, and recoverable as immutable artifacts.
The next thing I'd recommend before investing heavily in IaC is auto-recovery. The most obvious example is Autoscaling Groups, but any health check combined with an automatic action such as restart or re-deploy can work. This works best with Immutable Infrastructure, and typically does not require IaC.
> This also means leaning heavily into all the service offerings and orchestration tooling that is afforded to you by your platform.
I've built a data lake and several ETL pipelines using AWS native services (Kinesis, Lambda, Athena). It works but it's a bit...fiddly. I spend a lot of time configuring these services and handling various failure modes. I've been wondering if I should be looking at third party vendors like Fivetran or Matillion for ETL.
Does anyone who's worked with AWS data engineering services have thoughts on the trade-off between AWS native services and third party vendors in this area?
It's also the most democratizable model I have seen - anyone who knows the slightest amount of SQL can be set up to explore the data in minutes.
The elephant in the room is that you need to use SQL. Their spark connecters are as of now useless, so you either have to go with DBT, some homebrew SQL stringing mess or something like sqlalchemy. We're currently developing some wrappers around sqlalchemy to make this a bit less painful, but it's still so worth it.
I tend to separate all of my projects/services, and each of those to environments.
A cold storage AWS account, audit and security (ship logs, config changes, etc), shared services to another account.
If dev account gets hacked, that sucks, but we can clear it out.
It prod gets hacked (and deleted!) that super sucks. But hopefully cold storage and audit accounts can help us out.
If some other services/projects account gets hacked, I don’t want to be worried about impact to unrelated projects.
If you're going to do that, though, you may as well do CloudFormation into customer VPCs and let them do all the "paying Amazon".
It's all good advice, but please provide me with the money, and resources necessary to do it all.
Specific critiques:
> #1 Build as if you may sell at any time
> ... it forces you to build with best practices and isolation.
The opposite. It gives you an incentive to postpone technical debt, you want to grow and be acquired at the expense of whoever is going to integrate your startup into $bigco later.
Side note: "AWS Organizations" to me is simply a way for AWS to try to cover for the poor design choices of the organizational structure of an AWS account, and the unnecessary complications related to billing and metering.
Try to understand the AWS bill of a sufficiently large company - you won't. The AWS rep won't. The AWS Solutions Architect won't.
Also, never heard of a company being acquired at a higher price because they had a "proper" AWS setup.
Ah, forgot this: if your acquirer is using MS Azure, good luck telling them that you are using Cloud9 or other AWS-specific stuff.
Let's continue...
> #2 Build as if you may open-source at any time
Yes. In an ideal world. In practice, almost nobody follows best practices. Because there's always some urgency that takes precedence. This is why companies like Accenture, PwC, Deloitte, etc, keep billing monstrous amount of money to help large companies "migrate" or "evolve" or "adapt" or whatever buzzword they use.
> #3 Build with a cloud-native mindset
> ... going outside of the platform should be an exception and something you do only when truly needed
> My thinking on serverless these days in order of consideration. > - If the platform has it, use it > - If the market has it, buy it
Serverless, really? A promising, cutting-edge technology, sure. You want to bet your startup on Lambda? Go ahead. Lambda has been around for ~6 years now (I even tried a super early version internally before it was released), and I still haven't seen a large company doing A LOT of development on Lambda. Most project using Lambda are small, confined, isolated projects and/or teams.
Cloud native is great, WHEN it makes sense. I don't like religion too much, and I don't like religious people either; the author seem to have taken his faith in AWS too seriously.