Operations magic cure: nightly server restarts
blog.aparicio.org
blog.aparicio.org
It's not the act of restarting the servers that solves problems. Rather, their nightly reboot policy forces them to have all of the infrastructure in place to be able to rotate servers in and out of service quickly. It forces them to centrally manage configuration. It forces them to architect their software and networks to handle restarts.
They use server restarts as a way to force all of these things to be in place, which isn't such a bad idea. More importantly, it doesn't sound like they use the nightly restarts as a substitute for actual troubleshooting (e.g., rebooting a server to 'fix' a problem, rather than figuring out what's really going on).
Being an Erlang fan, I also recognize some other aspects of high-availability strategies such as:
* http://en.wikipedia.org/wiki/Crash-only_software * http://en.wikipedia.org/wiki/Microreboot
I also find it relatively unbelievable that the linked post claims that restarting servers requires a 24/7 operations team. He trusts his servers to do a bunch of other complex operations unwatched but doesn't trust them to do a restart? Obviously you need to make sure the next server waits until the one before it in the schedule comes back up before it dives down and send you an alert if one dies and doesn't come back, but otherwise this isn't something someone should have to run by hand.
These aren't mainframes, you don't need an operator anymore.
Have one if you want but you sure shouldn't need one.
On apps that cache database objects in app server memory, we see a noticeable degrade of response time while the app server re-caches database objects.
The opposite of this is the "server that must not be rebooted EVER" (and if it has to be will require the whole engineering team -I've seen one or two of these in my time). Likewise machines that are rebooted every night soon turn into "that's the way we've always done it".
So - it can be ok to fix things like this sometimes but watch it doesn't become a cargo cult fix.
It's just poor management of servers. If you have a process that routinely leaks memory or stalls then manage that by restarting it & balancing it but rebooting the entire machine! This implies that you don't trust your OS to NOT leak resources (pipes, files, sockets, whatever) thus you need a new OS.
The 'excuse' that it forces you to build a scalable redundant system is bogus, it's folly.
I've got a mantra that I have to repeat pretty often:
"Re-booting is not a valid troubleshooting technique."
Grrr... Till we sort _that_ one out, it's gonna be regular reboots...