Department of Redundancy Department

I got a call (well, a text, really) this morning, prompting me to hurriedly run out of my house and head to the office. My linux server, the workhorse, the reliable one, was apparently offline. Suck.

I get there, and while it is powered up, it is absolutely unresponsive. Might as well had been off. So, I powercycle it, and up it boots. Unfortunately on the front of the case is a orange blinky light. As it turns out, one of the two Ghostbusters-esque power supplies (PSU) (hooray, redundancy!) is dead. Boo, hiss.

Because there are two PSUs in the rig, it just pulls all the power from the other, good, PSU. That’s all fine and dandy, unless there is a power hiccup.

“But wait!”, I hear you cry, “What about UPSes? Don’t you have battery backups?” Very good point, young reader. I indeed do have UPSes. Two APC Smart UPS 1000s, to be exact. And to be extra redundant, one PSU from each server is connected to each battery, instead of each server getting its own dedicated battery. This way, if a UPS fails during an outage, one battery can still provide power to both servers.

Back to the power hiccup. Apparently, one of the UPSes died recently. Not a biggie, right? That’s why I mixed the power sources. Normally, it would not be a big deal, unless THE REMAINING GOOD PSU IS PLUGGED INTO THE BAD UPS!

Yeah. So, in the interest of visual simplicity, the setup is below:
GREEN = Power, battery backup.
ORANGE = Power, no battery backup.
RED = Dead

Not so UPS