Hacker News Reader

Comment by js2 | Hacker News Reader

It's 1995 or so. I'm at U.F. getting my CS degree. Our facilities guy is giving a tour of the department's server room to some big wigs.

For some reason, he decides to demo the UPS cut-over switch. I have no idea why. But he manages to toggle it the wrong way and instead of switching the entire room full of servers to the UPS, he manages to cut power to _all of them_.

My recollection is that the cooling went out too and the room was suddenly very silent. But in retrospect that doesn't make sense.

What I do remember is that it was non-trivial to bring all our Unix servers back up because over the years they had been setup with NFS mounts in a loop such that for A to boot, it needed B to be up, which needed C to be up, which needed A to be up.

Oops.

So it took a lot of manual intervention to bring everything back up.

View on HN

I was in that room! Poor fellow. It was the demo of our new UPS; I can still remember the gradual fading fans and clicks of capacitors.

He was really nervous showing the new install off to us, and he was just talking through the positions of the switch; but he actually moved the switch to each of them as he did it.

Back in the days the conslops ran the asylum. :)

Any old school UF types looking at this, The Dog list lives on, and is again meeting from time to time. :) But at quieter venues. Our ears have unaccountably gotten old.

The heck is a conslops besides the food at the prison?

Consultant / operator.

We were student employees, who were nonetheless in charge of all the systems. It was an absolutely awesome experience.

At least as valuable as my actual CS classes.

Console Operators

s/capacitors/relay/

Sorry, couldn't hold myself :)

Vaguely related, in 2015 we were building a new platform based on Cisco UCS and NetApp filers.

Cisco had a virtual router. N1000 perhaps, the hardware was a C220 and it had some sort of appliance running on it. Which depended on some sort of LUN from NetApp for its storage. But you couldn’t stand the LUN up until the vCenter was up because UCS provisioned the LUN, it didn’t work if you did it manually, and that ran on vCenter, and the vCenter depended on the router so that it could reach the LUN. It was pure circular logic hell.

There was a path to make this work, but it literally took half a dozen very ~~clever~~ expensive Cisco and NetApp engineers a couple of days in a room with a whiteboard to figure it out. It was absurd.

I've got two such stories. Around 2000, I was working at a huge web hosting company (third shift, monitoring the network). Suddenly the power went out. Turns out, the building management (separate company) decided to run a UPS test and the memo got lost. Fun time that night.

Second story, around 2005---I, along with a friend and my father, were in Las Vegas eating lunch at one of the major casinos when the power went out completely. It was eerily silent and dark! (and then slowly, we started hearing the groans of slotzombies rising among us) I'm sure someone lost their job for a UPS cut-over failure.

We had a blackout (seemed to happen every fall when the students arrived - the university had a power station), the NFS server needed NIS to boot, and NIS server required NFS. We managed to manually get the NIS service running in single user mode, brought up everything else, and then rebooted NIS.

I remember a day when a disgruntled employee shut off an entire Exodus data center by slapping the EPO switch. Took a long time and a lot of work to get everything in our cage back up.

Reply on news.ycombinator.com