Comment by mechanical_fish | Hacker News Reader

Comment by mechanical_fish | Hacker News Reader

Parent Full thread

mechanical_fish

I kind of liked the hardware refresh policy. It's not necessarily because the hardware goes stale. It's to keep the process and the personnel from going too stale.

By constantly mixing in new hardware one piece at a time, you compel code to be runnable on multiple generations of hardware at one time, avoiding flag days; you continuously shake the bugs out of new code, preventing it from growing a hardware dependency in year N that only gets discovered in year N+2; you periodically drill the team (especially the newer folks) in the procedure for bringing up new boxes but also accomplish real work (gradual upgrade of the server farm) in the process; you'll end up running hardware with a continuous range of model numbers and batches, perhaps mitigating against flaws that strike entire batches at once; when disaster strikes and you have to replace a box ASAP, odds are better that you've set up a similar box in recent memory and know exactly what to get, how to set it up, and what any pitfalls might be - and if there are pitfalls, you discovered them during working hours on spare hardware, rolled back, and spent a few weeks fixing them instead of discovering them at 5 AM on a Saturday morning and having to fix them on the fly.

(Of course, my devops team does everything in AWS, so what do I know about managing hardware?)

I agree that this checklist is way too long to vie with the Joel Test, though.

What you describe belongs to what is called "platform spread" and is generally something to be avoided.

What mechanical_fish describes is totally normal, and the result of normal growth and hardware refreshes. A company doesn't start with 0 servers and buy 200. They start with 5, then add 15, then 30, then 50. After three years they get to 200. Then they start replacing them in waves. New generations of hardware become more efficient, you can actually save money by replacing old hardware.

The "platform spread" is inevitable. Besides that it is impossible to physically replace all your servers at once, the bean counters also prefer it to be spread out. They don't want to replace $1,000,000 worth of servers every four years, they would rather replace $250,000 every year.

Sysadmins and devs shouldn't really care if they have five generations of hardware. There are a few things that matter so new hardware fits into the infrastructure, like having IPMI, serial ports for console redirection, enough RAM to run the apps. Try to keep as much as possible the same, like it is way better to only have 3 models of spare power supply to keep in stock vs keeping 20 models. Beyond that I don't think it matters. If every server was a different model it would suck, but having 5 generations is normal and good.

That's interesting. Perhaps the fact that I've learned about ops entirely in the era of virtual machines running atop disposable, generic, and (in the case of AWS) entirely invisible hardware has distorted my thinking on this matter...

"Sysadmins and devs shouldn't really care if they have five generations of hardware."

I wish it was true. sadly there are some services where scale and latency is so carefully measured that individual software releases are rejected if performance gets worse (or unacceptably worse, etc). In these situations you need to test on all hardware platforms. It is much better to have fewer platforms: Optimally: the one you are migrating off of, the one you are moving to.

For desktops... have you ever tried to maintain an Windows or Linux desktop environment with more than 4 "standard desktop configurations"? It becomes a nightmare. If you have a single "gold image" you blast to all machines it makes the task harder; if you stay with the vendor's OS and try to maintain it "forever" it is even worse.

One thing that makes virtualization a "win" is that the virtual box looks like a single hardware platform. It reduces testing, etc. However, then you still need to test the virtualization software on all hardware platforms... so you've made things easier for everyone but that team.

Reply on news.ycombinator.com