Preparing for Disaster: The Katrina Lessons

Sigmond Solares and Donny Simonton shared with ISPCON attendees the lessons they learned from running a data center in New Orleans during Katrina.

by Alex Goldman ISP-Planet Managing Editor [May 22, 2006]

In September of 2005, when hurricane Katrina hit New Orleans, many in the tech community were following the situation on the blog of the data center that runs directNIC. At ISPCON last week, Sigmund Solares, the owner of the data center and the CEO of Intercosmos Media Group, and Donny Simonton senior vice president of software development, gave a presentation to ISPs, intending to help others learn the easy way what they had learned the hard way.

The data center was in the center of town on the ninth floor of an office building. Adjacent to the office building was a car park, which would allow those who stayed to drive up and down the nine floors when they needed to. The data center had been built by Enron and Solares said he bought it in 2003 "for five cents on the dollar."

Prepare your people
Simonton said that when the disaster hits, your employees are not working 100 percent. They're worried. "Everybody's freaking out, watching TV. Our policy is that if you think you need to leave, go."

Also, the disaster doesn't necessarily hit at the perfect time (see Katrina timeline). For Simonton, it hit during the weekend he was writing up employee reviews. While he wanted to know which employees were staying or going, they wanted to know the size of their raise as a percentage of current salary.

Solares said that there had been so many near misses that he didn't do some things that he normally would have done to prepare for potential disaster. "We used to buy food. Not just regular sandwiches, but gourmet food. But we didn't do this for Katrina."

In addition, the company had plastic tubs for water, but did not fill them. They used toilets on a different floor each day. "Don't use the toilets on the floor you're on," said Simonton. For showers, put a water cooler on top of a cabinet—that's not much of a shower.

The company had ten air mattresses to sleep on, but did not have an air pump, so people took sofa cushions and duct taped them together. Lesson for the future: have much more duct tape on hand than you ever think you'll need.

Solares said that that particular day, he forgot to wear his belt. "I used an Ethernet cable," he said.

Solares added that he had not refueled his car that day because "gas lines were too long."

The company had a truck that was to be used for hauling barrels of diesel in an emergency, but, Solares said, his partner's mechanic decided that it was the ideal vehicle in which to evacuate.

The company had a positive relationship with the building supervisors, who left them a skeleton key for the building.

The company wishes they had stored more air fresheners before the emergency.

Solares also said that many who said they would come to help out at the data center didn't. One very useful person who was present but not part of the company's plan was Solares' friend Michael Bennett, ex-Special Forces, who wrote many of the blog entries and was useful in dealing with the military (such as when the 82nd Airborne woke up everyone in the data center with a gun pointing at their heads).

No power means no lights
The data center crew eventually ran Christmas lights down the stairways, but did not run lights in the parking garage. Instead, they eventually knew the route and were—literally—able to drive up and down the garage blind by the end of the ten days they were without power.

No power means no communication
Cell phones stopped working, although text messages did get through.

Some people had evacuated without their laptops, and others had never needed laptops before. "We allowed our employees to go into CompUSA and just buy a laptop if they needed one," said Simonton.

One person, Chris Anderson, who works the night shift, arrived for his 9 PM to 6 AM shift completely unprepared for Katrina. He'd been asleep all day and had neither listened to the radio or TV warnings nor had he checked his e-mail to receive the warning that Solares had sent out to all employees.

The blog had been set up casually (the only entry for Saturday August 27th says, "Hmm. This could actually be a nasty storm."). However, the blog proved vital for communication. Simonton said, "I was leaving lots of phone messages, and I started telling people to go to the blog."

Not everyone realized that Solares wanted to hear from them. "Make sure your people know the importance of contacting you," he said.

One unexpected good story—direct deposit continued to work, for those who had left the disaster area. Salaries were still paid.

Cash is useful
Even though people outside the disaster area were being paid, there was no way to obtain cash in New Orleans for the ten days the power was down.

"One guy needed cash to evacuate," said Simonton.

People do what you tell them to do
The company had a check list that employees followed. The check list said that the emergency oil tank was full, as were the drums. They weren't. Since Katrina, the check list has become more specific. It's not just enough to open the lid and see that there's oil inside. To fill out the checklist, you have to measure how much oil is in the emergency tank.

"Guaranteed delivery contracts"
Solares said that many contracts that guarantee repair and delivery have disaster escape clauses. The data center's UPS system needed attention. Alarms were going off, and the team at the data center did not know why.

Liebert first insisted that the technician be vaccinated. Once he was vaccinated, he would go to the checkpoint each day and get turned back. At that point, the data center was part of the relief effort, so the technician simply had to call Solares and he would have been able to get the technician through the checkpoint. Eventually, he did call Solares and did get through to the data center.

Conclusion
Now that even New England has what looks like a tropical monsoon season, everyone needs to be prepared for disaster, and whatever the cause of the disaster (hurricane, tidal wave, earthquake), the damage will be done by water and fire, as it was in Katrina.

— End