Storage Area Network (SAN) Maintenance

Maintenance Window:
Tuesday, July 22, 2008 21:00 - 22:00 PDT
(Wednesday, July 23, 2008 04:00 - 05:00 GMT)

Customer Impact:
The Storage Area Network (SAN) and managed backup services will become unreachable for up to two minutes. However, due to the retry mechanisms built into the iSCSI protocol, your applications should simply I/O block until the service comes back online.

Maintenance Details:
Due to complications regarding an unrelated issue, it was  necessary to postpone this maintenance by a week. In order to improve the stability of our switching platform, ServePath will be migrating your individual cross-connects between our Private Switch and Backend Services Networks to a high-capacity trunk.

If there are any additional questions before, during or following the maintenance, please direct them to the ServePath Support Center at +1-866-321-PATH (7284) or 1-415-869-7000, or open a case on-line at https://my.ServePath.com.

RFO has been delivered to Affected Customers for 3/13/08 Private Network Outage

A Reason For Outage (RFO) has been emailed to all customers who were affected by the Private Network Outage of 3/13/08. If you were an affected customer and have additional questions, please open a Support Case through your Customer Portal.

Again, we apologize for the inconvenience that this may have caused you or your users.

NOTICE - Please Call ServePath Support if you Continue to Experience Private Network Issues

While we have put in extensive network troubleshooting and engineering hours and believe that most Private Network issues have been resolved, there may be some customers still experiencing latency and/or other back-end network connectivity problems.

If your servers or networks are currently experiencing issues, we ask that you contact ServePath Support at:
866-321-7284 (415-869-7000 for international users) 
and press
Option 2
to speak with a Support Representative immediately.

We appreciate your extended patience and understanding during and after this outage that affected some ServePath customers.

[This was published on 3/14/08 @ 10:00 am Pacific Time]

Resolved -- Private Network Outage

*** RESOLVED at 23:30 PST on 3/13/08 ***

The outage has been resolved and all affected customers should be seeing systems operating normally. All affected customers have been notified via email and/or phone of the outage and resolution. Please note that maintenance and monitoring is still happening on those systems that were affected by this. Customers may be contacted for additional information.

More information (RFO) will be forthcoming shortly.

We sincerely apologize to the customers that were affected by this issue.

*** PARTIALLY RESOLVED at 20:50 PST on 3/13/08 ***

ServePath engineers have been resolving the issues with the private switch network and many customers who were affected are now back on-line. Please note that this outage affected only a small group of ServePath customers.

There are still some affected customers who may not have their service fully restored yet.  Rest assured that we are doing our utmost to resolve this issue and bring these remaining customers up.

We continue to monitor the environments, taking any pro-active measures as needed. More detailed information will be coming, including an official Reason For Outage (RFO), within the next day.

*** Updated at 16:50 PST on 3/13/08 ***

We have identified a need to replace hardware on the back-end private network and are currently performing this work with hardware that was on hand for a future network upgrade.
We are doing all in our power to resolve this situation as soon as possible. A detailed RFO will be forthcoming once the situation is stabilized.

*** Updated at 16:00 PST on 3/13/08 ***

ServePath engineers continue working towards a resolution of the private switch network issue. Please continue to check this entry for the latest information.

*** Updated at 14:15 PST on 3/13/08 ***

ServePath and Cisco technicians have identified and isolated the private switch network issue currently affecting some ServePath customers and are currently taking steps to expeditiously resolve the issue.

This post will be updated regularly with any important information as it is obtained. A full Reason for Outage (RFO) will be provided as well.

*** END OF UPDATE ***

The ServePath Data Center facility is currently experiencing problems within the private switch network. ServePath's network engineers are working with Cisco to attempt to resolve the matter presently.

Customer Impact:

Customers with a privately addressed vlan (non-public IP addresses), Cisco ASA firewalls, Netscreen 5GT firewalls, Sonicwall firewalls, and multi-server architectures behind one public switch port may be experiencing packet loss or high latency.

COMPLETED -- Scheduled Power Maintenance: March 13, 2008 @ 08:00 PST

*** COMPLETED - 16:10 PM Pacific ***

Transfer back to the power utility is complete and ServePath/ColoServe is currently running under utility (PG&E) power. No incidents of losses of power within the data center were reported while equipment was running on diesel generators or UPS systems.

**Update @ 09:42 PST**

Power maintenance is presently under way, and the data center facility is running on generator power.

The ServePath Data Center facility will be running on generator power for up to 9 hours while Pacific Gas & Electric performs maintenance to the power supply lines on the ServePath street.

Maintenance Window:

Thursday, March 13, 2008 from 08:00 PST - 17:00 PST
(Thursday, March 13, 2008 from 16:00 GMT - 01:00 GMT)

Customer Impact:

There should be no impact to the power in the facility and no down time is expected. This notice is for advisory purposes only.

Additional Details:

The power utility PG&E will be performing maintenance on electrical service lines in the streets adjacent to ServePath's data center facility in order to accommodate the power needs of a large new building adjacent to our facility.

As with any maintenance there are potential risks, but this is an advisory notice only as we do not expect any issues. If there are any additional questions before, during, or following the maintenance, please direct them to ServePath Support Center via phone - Toll-Free 1-866-321-PATH (7284) / International +1-415-869-7000 - or open a case on-line at https://my.ServePath.com .

Scheduled F5 Maintenance: March14, 2008 @ 00:00 AM PST

Maintenance Window:

Friday, March 14, 2008 from 00:00 PST - 01:00 PST
(Friday, March 13, 2008 from 08:00 GMT - 09:00 GMT)

Customer Impact:

There should be no impact to customers. This notice is for advisory purposes only.

Additional Details:

The operating system on the F5 Global Traffic Manager will be upgraded. ServePath is performing this proactive maintenance to upgrade to a more stable release of the F5 GTM.

As with any maintenance there are potential risks, but this is an advisory notice only as we do not expect any issues. If there are any additional questions before, during, or following the maintenance, please direct them to ServePath Support Center via phone - Toll-Free 1-866-321-PATH (7284) / International +1-415-869-7000 - or open a case on-line at https://my.ServePath.com .

Completed -- Scheduled Fortinet Maintenance: Mar 12, 2008 @ 9:00 PM PST

**Update @ 10:42 PM PST, 3/11/08**

The scheduled maintenance went as planned, and the Fortinet is showing expected version information.

Maintenance Window:

Tuesday, March 12th, 2008 from 9:00 p.m. - 10:00 p.m. PST
(Wednesday, March 13th, 2008 from 5:00 a.m. - 6:00 a.m. GMT)

Maintenance Details:

ServePath will be performing a proactive upgrade of the Fortinet firewall to a more stable release of the FortiOS.

Customer Impact:

During this second maintenance, your server may be unavailable for up to 5 minutes.

Contact Information:

If there are any additional questions before, during or following the maintenance, please direct them to the ServePath Support Center at 1-415-869-7000 or open a case on-line at https://my.ServePath.com .

Completed -- Scheduled Phone Maintenance: March 7, 2008 @ 6:00 PM PST

**Update @ 11:18PM PST, 3/8/08**

The scheduled maintenance was a success. All phone service has been restored.

Maintenance Window:

Date: Friday, March 7th, 2008
Time: 6:00 pm - 8:00 pm PST / 3:00 am - 6:00 am GMT on 3/8/08

Maintenance Details:

ServePath will be migrating the Phone System from one location to another. This is the second of 2 maintenance windows for this task. The first was completed successfully on Wednesday. The one listed above is for the full cut-over of the equipment (from T1/PRI circuit provisioning to the porting of numbers).

Customer Impact:

During this second maintenance, all inbound calls will be pointed to our answering services. Users with Support Issues are encouraged to log into http://my.servepath.com and open a ticket for any issues during the maintenance.

Callers with Sales concerns are advised to call the following day during normal Business Hours and will be given the option to leave a message on the Sales Voicemail box.

Callers with Billing concerns are advised to call the following day during normal Business Hours and will be given the option to leave a message on the Billing Voicemail box.

Contact Information:

As specified above, users with Support issues should log a ticket on http://my.servepath.com . Users with Billing and Sales issues should either call back the next day during normal Business Hours or leave a message in the respective voicemail boxes.

Completed -- Scheduled Phone Maintenance: March 5, 2008 @ 6:00 PM PST

**Update @ 10:26 PM PST, 3/5/08**

The scheduled maintenance was a success.  There will be another similar maintenance window (part 2 of 2) on Friday, March 7th, 2008 from 6:00 PM to 8:00 PM PST.

Maintenance Window:

Date: Wednesday, March 5th, 2008
Time: 6:00 pm - 8:00 pm PST / 3:00 am - 6:00 am GMT on 3/6/08

Maintenance Details:

ServePath will be migrating the Phone System from one location to another. There are currently 2 maintenance windows for this task. The one listed above is for the Pre-Test of the equipment (from T1/PRI circuit provisioning to the porting of numbers are done correctly).

Customer Impact:

During the Pre-Test period, all inbound calls shall be pointed to our answering services as well as On-Call Support cell phones. Users with Support Issues are encouraged to log into http://my.servepath.com and open a ticket for any issues during the maintenance.

Callers with Sales concerns are advised to call the following day during normal Business Hours and will be given the option to leave a message on the Sales Voicemail box.

Callers with Billing concerns are advised to call the following day during normal Business Hours and will be given the option to leave a message on the Billing Voicemail box.

Assuming everything tests positively, final migration will take place on Friday, March 7th, 2008 from 6:00 pm PST to 8:00 pm PST.

Contact Information:

As specified above, users with Support issues should log a ticket on http://my.servepath.com . Users with Billing and Sales issues should either call back the next day during normal Business Hours or leave a message in the respective voicemail boxes.

Completed -- Scheduled Backend Switch Maintenance: Feb 29, 2008 @ 7:00 PM PST

**Update @ 9:39 PM PST, 2/29/08**

The scheduled maintenance revealed that the switch-port module was not defective, but the switch chassis. This event will be re-scheduled and the chassis replaced at a later date to be determined.

Maintenance Window:

Date: Friday, February 29st, 2008
Time: 7:00 p.m. - 8:00 p.m. PST / 3:00 a.m. - 4:00 a.m. GMT

Maintenance Details:

ServePath is replacing a defective switch-port module on a backend customer switch. ServePath is performing this proactive maintenance to prevent an unexpected outage from occurring.

Customer Impact:

There should be no impact to public connectivity, but there will be up to 5 minutes where backend connectivity will be interrupted.

Contact information:

If there are any additional questions before, during or following the maintenance, please direct them to the ServePath Support Center at 1-415-869-7000 or open a case on-line at https://my.ServePath.com .

Completed -- Scheduled Power Maintenance: Feb 22, 2008 @ 7:30 AM PST

**Update @ 2:45 PM PST, 2/22/08 **

The following scheduled maintenance was completed at 1:35 PM PST Maintenance Window:

Friday, February 22, 2008 from 8:30 AM PST - 11:59 PM PST
(Friday, February 22 4:30 PM GMT - Saturday, February 23 8:00 AM GMT)

Details:

ServePath's datacenter facility transferred to generator power at 9:03 AM PST. During a testing phase prior to the utility power being cut, we experienced an alarm on our primary generator. Although power to the facility failed over to our secondary generator, the maintenance was canceled because we did not want to risk operating on a single generator without backup power from the utility.

There were no interruptions to power for our facility. During the maintenance window, our data center switched between utility and generator power three times without incident.

The generator fault was identified as a malfunction with the roof cooling tower and has been remedied.

The power utility, PG&E, still needs to perform maintenance to the electrical lines in the street, and so the maintenance will be need to be rescheduled.

We will provide an update once the maintenance is rescheduled.

If there are any additional questions before, during, or following the maintenance, please direct them to ServePath Support Center via phone or open a case on-line at https://my.ServePath.com

**Update @ 12:36 PM PST, 2/22/08 **

Power maintenance has been canceled and we will be re-scheduling this event with PG&E in the future. All customers will be notified of the new maintenance window when it has been re-scheduled.**

**Original Notice**

The ServePath Data Center facility will be running on generator power for up to 15 hours while Pacific Gas & Electric performs maintenance to the power supply lines on the ServePath street.

Maintenance Window:

Friday, February 22, 2008 from 7:30 AM PST - 11:59 PM PST
(Friday, February 22, 2008 from 3:30 PM GMT - 7:59 AM GMT)

Customer Impact:

There should be no impact to the power in the facility and no down time is expected. This notice is for advisory purposes only.

Additional Details:

The power utility PG&E will be performing maintenance on electrical service lines in the streets adjacent to ServePath's data center facility in order to accommodate the power needs of a large new building adjacent to our facility.

As with any maintenance there are potential risks, but this is an advisory notice only as we do not expect any issues. If there are any additional questions before, during, or following the maintenance, please direct them to ServePath Support Center via phone - Toll-Free 1-866-321-PATH (7284) / International +1-415-869-7000 - or open a case on-line at https://my.ServePath.com .

Completed -- Scheduled Router Maintenance: February 21st, 2008 @ 9:00 PM PST

**Update @ 10:00 PM PST, 2/21/08 - the scheduled maintenance has been completed. All network activity has been returned to normal.**

Maintenance Window:

Date: Thursday, February 21st, 2008
Time: 9:00 p.m. - 10:00 p.m. PST5:00 a.m. - 6:00 a.m. GMT

Maintenance Details:

ServePath is upgrading the IOS on one of our border routers to a more stable release recommended by Cisco. This is the second and final border router upgrade.

Customer Impact:

While we do not expect any outage due to our multi-transit BGP architecture, your server may experience a brief period of routing instability for up to 30 seconds as BGP re-converges through one of our other transit providers.

Contact information:

If there are any additional questions before, during or following the maintenance, please direct them to the ServePath Support Center at 1-415-869-7000 or open a case on-line at https://my.ServePath.com .