37signals Leaves the Cloud
world.hey.com
world.hey.com
I think there's a third, which is when the company is so rich it just doesn't matter. Then saving even a little bit of employee time (which are more likely to be the bottleneck in such a company) is worth basically infinite spend.
I worked at a place where they spent $6k/month on cloud, just for a Java app not doing much plus a PostgreSQL database. I came up with a way to save $2k/month (better indexes), when I did it nobody cared or even noticed I think. I know that $2k/month might not sound like much, but I think the same principle would have been true even if it'd been much more. They spent about $2m/year on an external software development agency to create a React project, full of inefficient scrum meetings and billable hours from project managers and so on, and they didn't really care about that either.
Companies (particularly young ones) succeed or fail if they are innovating on their products and delivering value to customers. Reducing cloud spend is not doing that.
It may very well improve the bottom line, but when the company is (correctly) more focused on “how do we increase our customers by X% this year”, pursuing those savings is a distraction.
Then you end up with BS like the ones described here https://www.the-investing-desk.com/cloud-costs-are-in-a-bubb... where to justify the fact that cloud is much more expensive managers start to throw random numbers vaguely justified just to show that cloud is actually better.
I saw similar things in Corporate world even when choosing a colo/service partner vs doing in-house... managers asking us detailed costs for infra needed (servers, network, licenses etc.) and when you were getting something like 30 in-house vs 100 hosted they just threw some random FTE count to support it to get to 110 vs 100.
Another element of this is I would think its more fun for a systems person to be closer to the hardware. Specifying, ordering and unboxing servers, disks, switches and setting things up sounds satisfying to me.
As oppose to complexities of self-host using self purchased hardware?
You're not going to beat the reliability by going inhouse.
If you run a competent organization, this can be a boon. If you have serious deficiencies, it may well be the opposite.
Am I able to beat Amazon on their wide range of services? Absolutely not - if you're looking for FAANG-level scale, you're talking to the wrong person. But for the handful of services I do need (HTTP LB, app servers, relational database and cache), I beat them because I only need to deliver very specific services and can schedule the risky operations (maintenance, hardware upgrades, etc) around my business' needs, something AWS can't do because as a global provider any time they pick for maintenance would inevitably clash with someone's business-critical hours. Turns out most companies also only require a small chunk of what AWS offers, so the rest of the AWS control plane's complexity and its global scale is actually a major liability.
> AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy.
And yet that one time a client's RDS instance ran out of disk space, it got stuck in "modifying" for 12+ hours (I assume an AWS operator eventually manually logged in and fixed it) with no way for us to recover, forcing us to restore a backup and reconstruct missing data from logs to bring our service back online in a timely manner.
Would it fail the same way on bare-metal? Yes, but at least I can SSH in and fix it immediately. Not to mention, the only reason it ran out of space is because AWS' pricing forced us to be very conservative with storage, where as the bare-metal machines I rent - for far cheaper than AWS' offerings - come with terabytes of storage as standard so this incident would've never happened to begin with.
> The thing that Soul or other similar tools follow is to automatically turn your database into you backend. You don't need to implement any endpoints, Soul already did it for you.
may have no clue anyway how to be reliable - which is totally fine of course.
OP was talking about observability and diagnosability. These will be infinitely better/easier with access to the hardware & OS.
But on reliability, if you keep your stack simple and boring, you can pretty easily also beat them on reliability. Complexity is the top enemy of reliability (and security).
On AWS, you can have a Postgres instance with multi-region availability. How do you plan to do this with your own hardware?
Because I’d be dumb to spend hundreds of dollars and engineering complexity for a stable load service, when a 20 EUR VPS (10 EUR until recently - see, scaling!) runs everything I need (forecast proxy, geocoding, website, payment verification) just fine :).
(Link: https://apps.apple.com/cz/app/weathergraph-weather-widget/id... )
Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host.
Weird.
He also did not mention if they are going to self-host, colocate, or rent bare metal in a more traditional host (like softlayer). Probably colocate as I think that’s how they used to run?
Both your points are still valid though.
If it's $250k each, that's already $500k. Unless they can find someone who can do both for... $300k?
I don't know.
He doesn't provide enough details to make the math work.
Silicon Valley culture isn't nearly so critical or valuable that your organization requires it in order to do the necessary work.
Sure, it is not "that complicated" for basic stuff, simple sites, hobby projects or overall quick'n dirty solutions because you're under a budget, as with everything.
Doing infrastructure properly, with SLAs and a very high level of availability and redundancy (both in systems, and in people with the know-how) is incredibly complicated in my opinion.
I bet what they're going to save on the the Cloud provider is going to be spent (and probably even more) in salaries, if they keep the same levels of service.
Specially true considering they pay pretty good salaries.
This point gets raised every single time bare-metal comes up for discussion and at this point I'm not sure if people are legitimately unaware or if there are hidden agendas at play.
There are plenty of providers out there who will handle the logistics for you with an interface similar to AWS EC2 - ask for server and get SSH access in minutes/days/weeks (depending if it's an off-the-shelf or custom build).
From there, your existing DevOps guy (which you must have to navigate AWS' complexity) can take over.
It's only at either extremely large scale or very unusual requirements that managing your own hardware makes sense. Until then, you will never be able to beat the economies of scale of managed bare-metal providers.
I am sorry but how's it different from leaving BigCloud for SmallCloud? That's like saying, I'd no longer use Java/Kotlin because Oracle/Google but I'll however use Clojure because JVM.
Go back and re-read :)
They [believe that they] don't need to hire anyone. They already have a staff of ops folks that run their cloud operations. And those folks will run their on-prem operations.
If you're constantly installing stuff then you're scaling up a lot, and that's the level where it would make sense to have someone hired dedicated for it.
You can also rent hardware.
the thing is: cloud ops are also quite complex but it's a different kind of complexity. I don't know how many people are required.
The largest teams I've been on have been cloud deployments, and some of the smallest teams were bare-metal; but that's just an anecdote.
If you're "cloud native" you've already architected your stack to be horizontally-scalable and distributed (partly because the cloud is so expensive that running at peak load capability all the time would bankrupt you) so hardware failure should matter even less.
I think you already pay this guy to manage your AWS instances. so you get the benefit of the full 500k
Outside of expensive Silicon Valley and FAANGs, where do you get this $250k/Year ? I think most of us on HN are normalizing this whole "devs make 250k/year" thing when in reality, it is a tiny percentage of people. Tiny when you look at the entire ecosystem.
> Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS?
A company which has a total AWS bill of $500k/yr is a very small one (the article says 500K is just the RDS/ES bill, so total is surely much higher for them).
Our (tiny startup) AWS bill is ~200K/yr and the amount of compute and storage we do could very easily be handled by one single physical server.
Now, sure, if we went to self-hosted we'd split that into multiple servers for redundancy and growth headroom, but in any case definitely less than a dozen boxes.
You don't hire anyone to install and manage the hardware at that scale. Someone from the ops team goes to the colo for a day or two to set it all up and then you only need to go back when something needs replacement, which at that small scale is going to be very rarely.
That's exactly what we did at a previous startup and the company grew to many, many hundreds of people before there was so much infrastructure that it became worth thinking about hiring a dedicated person. By that time you'd be in the 10s of millions in AWS spend.
If the software is developed enough, this enables an even more global distribution. Software services don't want to manage machines and data centers don't want to manage customers.
Developers keep adding toys that cost money, and then at some point, someone at the top notices the bill, and it's all hands on deck.
The cost due diligence should happened before anything, not when your ass is on fire and you need to save money yesterday.
"Do we need that beefy RDS instance?" ¯\_(ツ)_/¯
"Why does our ElasticCache data keep growing at such a high rate and nothing is expiring?" ¯\_(ツ)_/¯
"Could X be done much cheaper with Y?" ¯\_(ツ)_/¯
"Why did we spin up DynamoDB for this if we already have Postgres?". ¯\_(ツ)_/¯
This is a good point. I really dont like the centralization on AWS.
So it's a matter of choice.
The only way of being really sure of what is going on is to own it yourself. Unfortunately that has become more and more important.
The executions we've had so far are terrible. There's huge markup, piles of complex abstractions, and all the other issues mentioned in this article.
When I was a teen, I used to pay this guy in town like $1.5 a month for a user on a *nix machine where I could run scripts and run my own IRC bouncer and stuff. A bunch of people shared this machine (mostly running IRC bouncers, AFAIK). I've no idea now many, it was stable, and nobody was eating up all the CPU. It worked.
I feel that this nostalgic memory of the past is closer to a utopic "cloud" than anything any modern provider is offering.
It strikes me that it seems like 37signals are just using AWS as a glorified virtual machine with maybe some db hosting. To me the big benefit of the cloud is using all the little services they provide, like s3, sns, sqs, CloudSearch ECS, cloudwatch and the list goes on and on. If we were to host all those service ourselves and keep them up to date the operating cost would be huge. Furthermore I work for a site that is very public and has a lot of for instance ddos threats hanging over head. AWS handles lots of ddos threats all the time and they a much better suited for handling that than our own ops team.
The microservices in AWS were feature-services that had their own datastores so only paid a single overall request latency than the many app-server to db queries per user request.
A whole load of Racks for running Basecamp? We are talking about 42U per Rack, and total of 420U of Servers?
The scale seems quite massive. At least to the idea / perception of what I had about Basecamp. Would be nice to see those specification and see how much of an improvement it is 10 years later and if we could fit those into 2-3 Racks.
Basecamp started on one single Rackspace server, before I was there. I started at 20 people, left at ~55.
When I left there were I don't remember how many racks exactly, but more than 10, less than 30 in the primary location. 42U of servers in each. There was a mix of a whole load of (Dell, never pay list price!) blade servers, DB appliances (~12 in total across ~6 apps ISTR), Isilon storage[0][1], F5 kit, juniper routers etc. etc. We had some epically fast storage in some of the servers for the time, way faster than SSDs.
Later we added two more sites. One in I think Virginia, one in NY. The one in Virginia was a replica of what we needed to run Basecamp, the one in NY was a half-rack data replication location (I think I got that the right way round). We had 10G fibre (we rented wavelengths not the whole fibre) between each location. We could lose one DC and remain RW for our block data, 2 DCs and we'd have to drop down to RO. Block data was things like uploads, so DBs, search etc. wouldn't have been affected. We could lose one of the /main/ DCs and still be RW for everything.
With all this kit we were able to run both main DCs hot. With our Geo DNS you could hit either of our DCs and you'd get served pages, you could even write to both locations. One DC was always the "RO" DC, it always replicated the databases. If you tried to write to that DC we proxied your request to the RW DC over our 10G links and proxied any more requests you made for n seconds to the RW DC too, at which point we reverted you back to the RO DC.
Now, NY to Virginia isn't that far, so why bother with the hot/hot config? Because it played into the rest of our plan, which was DC failover. With some pretty epic voodoo (Juniper/F5/OpenResty etc.) we could fail over the datacentres, swapping the RO and RW locations. We could also do this if one of the locations was unavailable. We could do this in 4 seconds /without losing a single in-flight request/ (we tested it).
This ended up a bit longer than I was intending, but it illustrates a few things:
- I think people underestimate Basecamp. It's /huge/ (money and users, not employees). Not so much in the tech world (anymore), but even with this kit, even with the (at the time 6) sysadmins that maintained it, it still made a shit load of cash. I guestimated the net-worth of the two owners as in the hundreds of millions of dollars each, entirely because of basecamp. Cash that as a privately owned company all went to the owners (who gave some of it to us, they treated us fairly well). I think it was Patio11 who said that people under estimated the market for software, it's easy to do, these aren't human-scale numbers.
- The owners are right about keeping it private. You might not make "larger yacht than Larry Ellison" money, but you sure as hell have a lot higher chance of making "pretty damn big yacht and not having to work again" sort of money. If I was rolling the dice I know which I'd gamble on.
- I don't think I ever actually calculated it, but the amount of money our servers were worth while running was /immense/. When you went to the DC and looked at the row of racks, you could kinda see just how dense the $ value there was. The efficiencies that the cloud brings gives money to AWS, not to the clients.
- All of this was still /way/ cheaper than running cloud infra. I'm not against the cloud (I use it), but we made more because we had this infra than if we were on the cloud. We were dense when it came to # customers per $ spent on infra.
- The flexibility we had because we controlled everything was also extraordinary. It came at a cost, we had to do it ourselves, but we had a /lot/ of power to make things work the way we wanted to.
- We had access to everything, from the network (hell, even the light :) ) to the JS. This gave us optimisation options not available to a lot of people. Network topologies, buying the right CPUs (we benchmarked them), configuring the CPUs, etc. etc.
[0] An Isilon storage engineer once dropped one of these on the floor, while it was powered on and the drives were running. It took out a metal floor tile. [1] They were a PITA, everyone hated them. There was a running joke among the sysadmins that when we decommissioned them we would take one on an Ops meet and use it as target practice.
Lot of people think that DC failover needs to be east coast / west coast but you can achieve most of your DC redundancy goals separated by 100 miles or less and have a lot lower latency, higher bandwidth and lower costs. Might want to think about different geographic flood plains and different power companies / grids.
Could still nerd fight about EMPs from nuclear war and a sufficiently massive hurricane or an earthquake out here on the west coast, but at some point you need to accept some risks.
Even if the number above does not include redundancy, that still only makes it 2 Racks of Servers with spare.
30 Racks is a lot. What am I missing here? Apart from Storage.
Bear in mind that we had I think 6 apps. Basecamp 1, 2 and 3 (all separate infra), Highrise, Campfire, some other internal stuff. Our ES cluster was pretty damn big, redis and memcached too. Juniper, F5, network switches (rack infra) etc. Storage was pretty big, quite a few 4u servers with spinning rust. The blade servers were I think 6 blades in 2u.
Everything was redundant, everything had 2x or more. I honestly can't remember how many racks we had. "More than 10" is my hand-wavey guess.
I don't think DHH was being disingenuous with his "2k on 30 servers" message, it was likely more about the scalability of Rails rather than the infra required to run the app.
It's also a huge company and a massive distributed system, so moving that off from AWS is an insane amount of effort and there might be legitimate benefits from AWS' management and autoscaling layer that aren't as valuable for 37Signals which runs simpler, monolithic systems.
It still cost /far/ less to buy/rack/power etc. our servers 100% of the time than it would have done to use the cloud and scale up and down with demand.
There is definitely a situation when bursty requirements will make cloud the more obvious answer, but given the near order of magnitude cost increase for cloud it's not as clear cut as just seeing a wave on a graph :)
2. Content Encoding are not done at a regular interval. And most of their work on ML and other quality requirements does not require fixed interval either. i.e They are schedule to work in Spot when AWS price are at its lowest.
3. Netflix get huge discount anyway.