Multiple servers offline - Data Center Evacuated - Last Updated 5/31/2008 - 10:23pm EST
A major wide-spread electrical problem in our Houston Texas 1 datacenter. It has been evacuated at this time and emergency services are on site. Unfortunately at this time we do not have an ETA to have it resolved.
At approximately 6:45 p.m. EST, a transformer in the H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers at this facility are down.
It has been determined that no servers have been damaged in the incident, but the servers are down because the power is still out at the facility. Techs are working to restore power and bring all systems back online.
Please note only clients located in this datacenter have been affected with this downtime. All other servers are online.
We will keep you updated. Thank you for your patience.
1
Datacenter Evacuation
Started by tshah, Jun 01 2008 02:45 AM
5 replies to this topic
#1
-
- Administrators
-
- 196 posts
Admin
- Gender:Male
- Location:Philadelphia, USA
Posted 01 June 2008 - 02:45 AM
#2
-
- Administrators
-
- 196 posts
Admin
- Gender:Male
- Location:Philadelphia, USA
Posted 01 June 2008 - 04:39 AM
Latest Update from The Planet CEO Doug Erwin:
"This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.
I am sorry that this accident has occurred and I apologize for the impact."
We sympathize with our hosting clients that are affected due this accident in datacenter. We will be available to answer any questions you may have, please be rest assured that all our support lines including phone lines will be responded promptly.
I would like to thank everyone for showing patience in this situation.
Thank you
"This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.
I am sorry that this accident has occurred and I apologize for the impact."
We sympathize with our hosting clients that are affected due this accident in datacenter. We will be available to answer any questions you may have, please be rest assured that all our support lines including phone lines will be responded promptly.
I would like to thank everyone for showing patience in this situation.
Thank you
#3
-
- Administrators
-
- 196 posts
Admin
- Gender:Male
- Location:Philadelphia, USA
Posted 01 June 2008 - 07:04 PM
Latest Update from datacenter:
"We expect to be able to provide initial power to parts of the H1 data center beginning at 5:00 p.m. CDT. At that time, we will begin testing and validating network and power systems, turning on air-conditioning systems and monitoring environmental conditions. We expect this testing to last approximately four hours.
Following this testing, we will begin to power-on customer servers in phases. These are approximate times, and as we know more, we will keep you apprised of the situation. "
We thank you for your continued patience.
"We expect to be able to provide initial power to parts of the H1 data center beginning at 5:00 p.m. CDT. At that time, we will begin testing and validating network and power systems, turning on air-conditioning systems and monitoring environmental conditions. We expect this testing to last approximately four hours.
Following this testing, we will begin to power-on customer servers in phases. These are approximate times, and as we know more, we will keep you apprised of the situation. "
We thank you for your continued patience.
#4
-
- Administrators
-
- 196 posts
Admin
- Gender:Male
- Location:Philadelphia, USA
Posted 02 June 2008 - 04:06 AM
Latest update from Theplanet CEO:
"As previously committed, I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.
As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.
There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.
We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.
Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.
We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.
We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed.
I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.
I plan to have an audio update tomorrow evening. "
We hope that we belong to PHASE 2 that is expected to be online by sunrise. Thank you for your continued support.
"As previously committed, I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.
As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.
There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.
We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.
Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.
We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.
We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed.
I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.
I plan to have an audio update tomorrow evening. "
We hope that we belong to PHASE 2 that is expected to be online by sunrise. Thank you for your continued support.
#5
-
- Administrators
-
- 196 posts
Admin
- Gender:Male
- Location:Philadelphia, USA
Posted 02 June 2008 - 05:31 AM
On further communication with the datacenter we have determined that our servers that are affected due to this accident are located in PHASE 2.
Thank you for your patience.
Thank you for your patience.
#6
-
- Administrators
-
- 196 posts
Admin
- Gender:Male
- Location:Philadelphia, USA
Posted 02 June 2008 - 09:43 AM
Fortunately, our servers were located in PHASE 2 and all of them are now online. We thank you for your continued support and appreciate your patience.
1 user(s) are reading this topic
0 members, 1 guests, 0 anonymous users
