existentia

I don’t know about the rest of you, but I noticed Vox (note: this post was written when the blog was hosted on Vox) was down for a period of time a couple of days ago and it transpired that this downtime was part of a much bigger incident. It was all over the news and apparently because of this power outage a number of popular web sites got hit and not just Vox. Some of the other web sites hosted in the same data center were other SixApart sites including LiveJournal and TypePad as well as the likes of CraigsList, Technorati, AdBrite, Second Life and Yelp.

Now, as most people who work in I.T. should be able to tell you, if power to a data centre is cut then it is reasonable to assume that they will have the appropriate contingency measures in place to keep power running during a power cut. This standby power is usually supplied by a combination of UPS devices and diesel generators.

So what happened in this case? Well, over the last two days details of exactly what happened have started to emerge. Valleywag is continuing to provide coverage of exactly what happened over here. Embarrassingly, it looks like that some of 365 Main’s (the data centre in question) diesel generators failed at the all important moment. I can see the lessons learned report already…

Personally, I preferred the rumour that did the rounds on the day in question. Namely, that a disgruntled data center employee came to work drunk and went on the rampage. Now how do you prevent something like that from happening!?