Root cause seems to be "server up too long."
I'd argue that a 5 why's analysis of "server up too long", leads to "server wasn't written well." YMMV
It's basically a scheduled sanity check for your infrastructure. It boils down to robustness and assurance. If parts of your system won't survive a reboot this is an unknown and therefore a risk.
If a system can't survive a reboot it's better to find that out when it's during a scheduled reboot and personnel is on hand to deal with it rather than relying on on-call personnel responding to alarms from monitoring at some point.
Apart from the repair time being longer when relying on on-call technicians, there is also the general observation that productive systems tend to become more valuable or have more exposure over time as their usage increases. A failure at a later point therefore also becomes more costly as time goes by.
Most of the time you don't need to do complete reboots. Restarting services usually takes care of the majority of the sanity checks.
Sure, that usually takes care of most checks, but the point is to eliminate as much as possible unusual scenarios. A service-level restart will catch service config mismatches, but it won't catch, for example, manual interface or routing table updates. It also won't catch the 'fragile root' problem where the device that the kernel is loaded from has become corrupt.
Bad things happen in the strangest places if you run enough systems.
But I agree - the objective is to detect those situations when things have gone wrong a long time ago and we just don't know it yet. ;-)
A common pattern is that each logical partition gets rebooted once per month, on a rotating basis. Say you have 4 LPARs: LPAR 1 is rebooted on the first week, LPAR 2 on the second and so on.
The goal is to demonstrate that in case of catastrophic failure that the system will come up. If it doesn't come back up during the scheduled reboot, you have the advantage that you are rebooting outside production hours and have some time to fix the problem minus frantic phonecalls.
Meanwhile, the other LPARs will keep running as if nothing is going on.
I guess bootstrapping is more common in the Lisp world.
Its a business decision. Not worth getting emotionally involved, just weight cost of fixing vs cost of rebooting.
Here's a hypothetical, but not ridiculous, scenario. Which is a better way to spend resources? Track down and stamp out an extremely difficult resource leakage bug? Or simply bounce the server? The costs associated with the former may be HUGE. The costs associated with the latter may be small in comparison (cost of potential down time, minimal labor cost of bouncing the server(s), and costs associated with user reliability).
I wouldn't always chalk it up to whatever it is that your comment implies (laziness? willful ignorance?).
It also looks really JV to see those alerts day in/day out. Like, come on. We can't just fix the problem?
Sure, we would all love to fix the bugs, but if there are no easy clues and a regular reboot fixes it - who really cares when there are features to build?
You begin to see why a microkernel that needs to be bounced more often than, say, Linux, which is not a microkernel, begins to lose some of the appeal of having a microkernel in the first place.
(Aside: Mac OS X is based on Mach, but practically all of the stuff applications see is provided by a single server which (to the best of my knowledge) can't be restarted without taking down the whole system. It's a hybrid design.)
Also, just to confirm a stereotype, do you use Windows for your servers?
Related tangent: There was a time when the the US military's COTS (Civilian Off The Shelf) initiative resulted in AEGIS missile cruisers running largely on machines running Windows NT. Yes, the ships responsible for screening carrier task forces against attack -- had to be rebooted on a weekly schedule.
To be fair, the unmolested NT kernel is rock solid. I know people used to run RAID arrays with it, but for the desktop, Microsoft allowed the crossing of certain architectural lines.
which, if memory serves me right, directly translates to "OS/2 kernel is rock solid"
"Given these issues [with OS/2], Microsoft started to work in parallel on a version of Windows which was more future-oriented and more portable. The hiring of Dave Cutler, former VMS architect, in 1988 created an immediate competition with the OS/2 team, as Cutler did not think much of the OS/2 technology and wanted to build on his work at Digital rather than creating a "DOS plus". His "NT OS/2," was a completely new architecture." http://en.wikipedia.org/wiki/OS/2
This is unsurprising: They're all designed by the same guy, Dave Cutler.
Too bad it resents running programs on it so much...
When that happens, it doesn't resent those programs so much as that it's been violated and turned into a warped version of itself. Sort of like Jeff Goldblum's character in The Fly.
So 3.0 was adopted, with success, but not as a microkernal hosting user-space servers.
That's what you get when you assign the management of Solaris servers to MCSAs...
Bouncing a server may even make the problem go away, but it does little to enlighten you on why it happened in the first place.
And yes, If your MCSAs tend to bounce servers all the time, that's because their experiences have shown this works for Windows.
You have a human resources problem, not a technical one.