Thursday, March 13, 2008

Resolved -- Private Network Outage

*** RESOLVED at 23:30 PST on 3/13/08 ***

The outage has been resolved and all affected customers should be seeing systems operating normally. All affected customers have been notified via email and/or phone of the outage and resolution. Please note that maintenance and monitoring is still happening on those systems that were affected by this. Customers may be contacted for additional information.

More information (RFO) will be forthcoming shortly.

We sincerely apologize to the customers that were affected by this issue.

*** PARTIALLY RESOLVED at 20:50 PST on 3/13/08 ***

ServePath engineers have been resolving the issues with the private switch network and many customers who were affected are now back on-line. Please note that this outage affected only a small group of ServePath customers.

There are still some affected customers who may not have their service fully restored yet.  Rest assured that we are doing our utmost to resolve this issue and bring these remaining customers up.

We continue to monitor the environments, taking any pro-active measures as needed. More detailed information will be coming, including an official Reason For Outage (RFO), within the next day.

*** Updated at 16:50 PST on 3/13/08 ***

We have identified a need to replace hardware on the back-end private network and are currently performing this work with hardware that was on hand for a future network upgrade.
We are doing all in our power to resolve this situation as soon as possible. A detailed RFO will be forthcoming once the situation is stabilized.

*** Updated at 16:00 PST on 3/13/08 ***

ServePath engineers continue working towards a resolution of the private switch network issue. Please continue to check this entry for the latest information.


*** Updated at 14:15 PST on 3/13/08 ***


ServePath and Cisco technicians have identified and isolated the private switch network issue currently affecting some ServePath customers and are currently taking steps to expeditiously resolve the issue.

This post will be updated regularly with any important information as it is obtained. A full Reason for Outage (RFO) will be provided as well.

*** END OF UPDATE ***

The ServePath Data Center facility is currently experiencing problems within the private switch network. ServePath's network engineers are working with Cisco to attempt to resolve the matter presently.

Customer Impact:

Customers with a privately addressed vlan (non-public IP addresses), Cisco ASA firewalls, Netscreen 5GT firewalls, Sonicwall firewalls, and multi-server architectures behind one public switch port may be experiencing packet loss or high latency.