Oct 17, 2008

Hurricane Ike and the Power of Planning

One of the toughest challenges when managing a data center is keeping everything up and running in the face of a disaster. Our clients depend upon us to keep their brand online and their back end operations working smoothly, regardless of the circumstances.

About a month ago, Hurricane Ike brought hurricane force winds to Louisville, leaving thousands of homes and businesses without power and an indefinite power outage to the home of our primary data center. This was the first major test of our emergency plan, and I am happy to announce we passed with flying colors. Power was maintained to the data center thanks to battery backup that supports 4 to 6 hours of power outage.  During that window, our plan went into effect, and our generators were fueled and put into action.

While I would love to say our plan was perfect, no untested plan ever is. One thing we had not anticipated was a fuel shortage created by such a large number of gas stations that were without power.  My fellow LEAP staff jumped to action and scoured the city to stock up a supply of fuel that would last us through the crisis.

The second challenge was the impact of the loss of non-essential power. The lack of main office lighting, air conditioning, and minimal power to the offices (generator power was dedicated to keeping the data center up and running) made the environment an impossible one for us to have full staff working on location. We were able to overcome this challenge with the assistance of friends who provided us with locations to stage temporary offices for our production staff. With the assistance of VPN (virtual private networks), we were able to function as if we were all in the same office, even while we were spread across town.

In total, our primary data center was without power for five days. As a precaution, however, we kept the generators running for another full day to allow the battery backup to gain a full day of charge. While it required all my staff to work almost 24/7 (we did take short naps in shifts), we were able to keep all client services running through the impact of the worst storm in 30 years.  This success, however, has not kept us from learning lessons that will make us even more prepared for emergencies in the future.