Stack Overflow: How we upgrade a live data center
blog.serverfault.com
blog.serverfault.com
The site has been a useful resource for so many years, and it works so well. It was a joy to discover that it all ran on like two racks worth of servers, and still does. Having seen corporate intranet portals, with maybe a thousand daily active users, running on excessive* hardware (needlessly, of course), it's like a breath of fresh air.
*EDIT: Removed hyperbole. Not more hardware, but too much nonetheless.
We obviously feel very different, and are just doing what works best for us.
And doing it very well, if I may say so.
however, i've seen very, very few people compare actual before/after $ figures on hn. when it comes time to show your cards everyone gets cold feet, either because they 1. don't have any idea because they aren't the ones paying for it or 2. don't have a baseline for comparison and are just paying whatever amazon asks or 3. it ended up costing 2-4x as much on amazon
when your bill is $5k/month 2-4x isn't that big of a deal. when it's $100k+ a month, it becomes a really big deal.
Upgrading hardware in EC2 is as simple as changing a launch configuration and updating an auto-scaling group. Maybe an hour of my time to update configs, verify and deploy. Updating something like a database or caching servers is more work for sure, but with 0 time needed to get to the DC, unpack, rack and configure servers you do save time with 'the cloud'.
I get that you do pay more for EC2 instances, especially if you keep hardware for 4 years. But AWS prices drop every year or two along with (generally) faster versions of software so your overall costs do drop.
How many ops employees would you need for a fleet of 500 servers in a datacenter? We managed it all with 4 people with AWS.
Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general the "scale horizontally" philosophy will favor the cloud while the "scale vertical" philosophy still seems to favor owned hardware in local datacenters. Which is superior is a crazy, long-standing debate with no clear answer.
✱cough✱ SQL Server licensing fees ✱cough✱
For what it's worth Basecamp also evangelize the "scale up" approach and they're on an open-source stack.
Scaling horizontally can let you use smaller, cheaper hardware on average and burst to higher capacity more easily if you need to, at the expense of a lot of complexity. It also (done right, which is rare) tends to gain you a greater degree of fault tolerance, since hardware instances become rapidly-replaceable commodities.
Most web apps have spiky but relatively predictable load. For example, a typical enterprise SaaS startup gets more traffic during work hours than on weekends. For these companies the complexity of developing a horizontally scaled architecture can be offset by the decreased cost of buying really big machines for peak load and then scaling back to a couple small instances for periods of below-average load.
That's (ostensibly) why AWS exists in the first place: Amazon had to buy a lot of peak capacity for Black Friday and Christmas and found it going unused the rest of the year. They never meant to sell their excess capacity, but they realized the tools that they built to dynamically scale their infrastructure were valuable to others.
Plus, a lot of work is offline data analytics, ETL, and so on. It's very cost effective to scale these workloads horizontally on-demand - spin up extra workers to run your reporting each hour/night and keep costs down the rest of the time when you don't need the capacity.
On the flip side, companies like Stack Exchange and Basecamp have high, relatively stable traffic worldwide. For companies like this it makes more sense to scale vertically - if they were in the cloud, they would never scale down or shut down their instances anyway.
Personally, I agree that horizontal scalability is oversold and most people can, indeed, scale up instead of out. However, a lot of plenty smart people disagree with me and have valid reasons to scale horizontally, too.
You still need to budget what you can get if renting dedicated hardware vs. renting virtual machines. For eg. a Dual Xeon X5670 machine w/ 96GB RAM and 4x480GB SSD can be had for $249 per month (just something random I found for demo purposes). Even if you do a reserved instance for a year on EC2, you can get a m3.2xlarge for this kind of money, and that's only 30GB RAM and 2x80GB SSD.
It might worth it to rent this sort of iron instead of spinning up and down EC2 instances especially if you can reasonably buy a large enough machine to cut a lot of headaches arising from distributed computing. The right tool for the job.
Owning hardware is again a different bag of hurt.
This could be a false dichotomy. Just because a service with AWS uses so many servers doesn't mean a more monolithic system would need as many.
We did talks with one of our competitors (before they were a competitor). We mentioned that we ran our infrastructure on 4 large VM hosts (with a light density of 3-4 VMs per host). They were shocked. They were currently running over a hundred EC2 instances, with the relevant satellite services. They literally could not believe that we could provide a comparable service without reliance on something like AWS.
It's amazing what an be done with the right knowledge. In our case, myself and one of my coworkers maintain our infrastructure at something like 4-6 hours per week total (mostly patching, reviewing logs, etc). We both have previous networking, hardware, and software experience. When we do major upgrades (about every 2 years), it takes one of us about a week to source the hardware, get it loaded in to the rack, and turned on. Then we migrate guests over and we're done. This doesn't even get in to the cost savings on running on our own hardware vs AWS pricing.
All own hardware, and having just had a reboot on Softlayer's schedule to fix the Xen issue for a separate project we're running on their gear - being able to schedule your own maintenance windows is so much nicer. We spend less time dealing with problems on our own hardware than we do dealing with cloud providers having issues.
I really don't understand the obsession of getting rid of ops / hardware guys and relying on Amazon/Google/CoolCloudProvider to handle everything.
After I left, the new CTO who replaced me migrated it Rackspace's cloud offering. I don't know the costs involved, but now the site averages an 8s load time.
You can't really beat the raw performance for the price of dedicated equipment.
I'd say our goal is to keep growing and serving more content without _needing_ 500 servers in a data center. We are doing pretty well at that so far. We'll see what happens in the future.
One day we got a new VP who came from a well known firm who was a "cloud expert". He moved (nearly) all of our infrastructure to AWS, after producing untold amounts of spreadsheets/power points expressing how much cheaper/better/faster it's going to be.
Long story short, it was 4x more expensive as running it in house. By the time they went back to our own infrastructure, most of the internal sysops(including "The Glue" guy) had moved on and much of the old internal hardware was re-purposed or gone. It was a fiasco that they still have no fully recovered from yet.
I would be very careful in characterizing AWS as the solution for every large scale computer infrastructure problem.
Conversely, I have had excellent experiences with AWS in my current job, although we still have a rather large HPC cluster internally which would never make sense to move to AWS.
Cloud is great, but there str still plenty of reasons to run your own datacenter. Yes for many startups it might not make sense, but at a certain sized company/application it can easily make sense.
The real upside of AWS is that they have relentlessly pursued and killed off reasons for you to care about things like this. They've eliminated points of failure in their infrastructure and given operators a wealth of tools to ensure their apps stay up through any update or event (AZ-affine ELBs and autoscaling groups, single-IP ELBs, continuous improvements to EBS and S3, etc.) Given the scale of their infrastructure in us-east-1, it's now also highly unlikely that any customer will manage to overload it on their own.
They are getting much better, as all providers are. They're still just not a fit for many people because of performance requirements that are either impossible or too costly to meet on that type of infrastructure.
I've always said this: the cloud isn't a good fit for us; do what works for you.
If you're going to bring that up, I can't resist reminding you of that time you had poor sysadmins running up and down stairs with buckets of fuel to keep your servers running[1].
> I've always said this: the cloud isn't a good fit for us; do what works for you.
There are costs. Frankly, it appears Stack Exchange prefers to lay those on its people rather than its purse.
[1]: http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-h...
When your neighbor's house is on fire you don't argue over the price of the hose. You help. Our remote people that couldn't come help in person also helped them replicate their entire network in AWS as a backup plan.
I don't usually post pissed off comments, but you're dead wrong here and intetionally or not demeaning a good company and good people whom, because they cared, came to help in a time of emergency. I take it you weren't in New York during Sandy; it looked like a post-apocalyptic war zone afterwards.
TL;DR - You don't know what you're talking about.
It's sad that you can't handle a snarky comment (right or wrong) that was given in response to your own snarky comment. If you can't handle being poked and it pisses you off, don't poke others.
Also, you seem particularly offended that your altruism is being maligned, when you're also complaining that the GP was unaware that the action was altruistic in the first place (???).
In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange. Articles like this make me so happy to be on AWS. Delegating this work to AWS allows us to focus on the product instead of the hardware that it runs on.
Actually it isn't.
Can AWS handle the scale of StackOverflow?
Absolutely. Netflix is a GREAT example of the scale AWS can handle.
However, can netflix migrate their entire system off of AWS without careful planning (which the parent of my comment was scoffing at)?
I'm a huge proponent of AWS, but I wouldn't be so naive to say that AWS allows me to be oblivious of hardware and infrastructure costs.
> In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange.
How so? StackOverflow is #56 in the world in Internet Traffic Rank (alexa.com).
We where hosted at that Peer 1 facility during Sandy. Also, if the water makes it up to the 16th floor it's time to give up anyway ... no one will care at that point :-D
For example, how failure scenarios handled and what is the role of slaves.
Other discussions:
http://meta.stackexchange.com/questions/69164/does-stack-ove...
http://meta.stackexchange.com/questions/110320/stack-overflo...
Slaves are not just for backups - they also serve as pub/sub mechanisms. Every publish propagates to slaves. Since we use pub/sub for things like web sockets, we can easily move that entire concurrent connection load to another data center with a simple DNS change. Yep, we've tested this - it worked well.
It is of course noting: redis just doesn't fail. We had one out of memory fail for the Q&A web sites when forking and that's it. It has been rock solid here.
Sigh, don't think it's gonna happen.
As a customer of Dell, it was interesting to see someone elses insight on managing a dell farm.
So neat! I am mildly envious! ;)
(EDIT: Over and above what's in the article.. Because they look quite nice, is why I'm asking - we haven't got any.. yet! Still performing well?)
The most annoying part was that the Force10 OS that the IOA's run is slightly annoying to work with when you are used to working on Cisco gear. I'd almost rather people stopped doing "it's close to Cicso" and did their own thing because the connotative dissonance is jarring when something is close but not quite what you expect.
But thanks (to both you and Nick) for responding, the FX2s look interesting enough that I might have to see about an eval unit :)
Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO?
I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.
Would be hilarious if someone who went to SO to figure out why the IIS splash was showing instead of his app happened to hit that page.
As always, we are looking how to adjust our deployment process to prevent it from happening in the future.
The next step in evolution is to have your script look for specific text on a page, which will change 24 months later and have all the perfectly good servers pulled out of rotation.
Can somebody explain why exactly? Was there ever some large scale research with the outcome this was somehow the cheapest? Or is it a mere byproduct of the consumption-based society? I'm asking because I'ved used lots of different types of hardware (both computing and non-computing) and >90% lasted well over 3 years. It seems a waste of time/money to get rid of ot after 3 years? Which matches with SE's findings: they choose 4 years. Still not that much, but already 'better'