Tuesday, March 25, 2008

Completed -- Emergency GoGrid System Maintenance

*** Update @ 21:38 PDT ***

Emergency maintenance took longer than expected but has been completed successfully.

Emergency Maintenance Window:

Tuesday, March 25, 2008 from 18:35  to 19:35 PDT

Maintenance Window:

Tuesday, March 25th, 2008 from 13:00 to 15:00 PDT
(Tuesday, March 25th, 2008 from 20:00 to 22:00 GMT)

Customer Impact:

Your GoGrid virtual server will be gracefully rebooted during this emergency maintenance period and will be periodically unavailable.

Maintenance Details:

We have identified a network configuration issue on some of our grid nodes that prevents some virtual servers from starting.  To resolve the issue, all virtual servers on affected nodes have to be rebooted to modify the configuration.  We apologize for this inconvenience.

If there are additional questions before,  during or following the maintenance, please direct them to the GoGrid Support Center at 1-415-869-7444 or open a case on-line at https://my.gogrid.com .

Friday, March 14, 2008

RFO has been delivered to Affected Customers for 3/13/08 Private Network Outage

A Reason For Outage (RFO) has been emailed to all customers who were affected by the Private Network Outage of 3/13/08. If you were an affected customer and have additional questions, please open a Support Case through your Customer Portal.

Again, we apologize for the inconvenience that this may have caused you or your users.

NOTICE - Please Call ServePath Support if you Continue to Experience Private Network Issues

While we have put in extensive network troubleshooting and engineering hours and believe that most Private Network issues have been resolved, there may be some customers still experiencing latency and/or other back-end network connectivity problems.

If your servers or networks are currently experiencing issues, we ask that you contact ServePath Support at:
866-321-7284 (415-869-7000 for international users
and press
Option 2
to speak with a Support Representative immediately.

We appreciate your extended patience and understanding during and after this outage that affected some ServePath customers.

[This was published on 3/14/08 @ 10:00 am Pacific Time]

Thursday, March 13, 2008

Resolved -- Private Network Outage

*** RESOLVED at 23:30 PST on 3/13/08 ***

The outage has been resolved and all affected customers should be seeing systems operating normally. All affected customers have been notified via email and/or phone of the outage and resolution. Please note that maintenance and monitoring is still happening on those systems that were affected by this. Customers may be contacted for additional information.

More information (RFO) will be forthcoming shortly.

We sincerely apologize to the customers that were affected by this issue.

*** PARTIALLY RESOLVED at 20:50 PST on 3/13/08 ***

ServePath engineers have been resolving the issues with the private switch network and many customers who were affected are now back on-line. Please note that this outage affected only a small group of ServePath customers.

There are still some affected customers who may not have their service fully restored yet.  Rest assured that we are doing our utmost to resolve this issue and bring these remaining customers up.

We continue to monitor the environments, taking any pro-active measures as needed. More detailed information will be coming, including an official Reason For Outage (RFO), within the next day.

*** Updated at 16:50 PST on 3/13/08 ***

We have identified a need to replace hardware on the back-end private network and are currently performing this work with hardware that was on hand for a future network upgrade.
We are doing all in our power to resolve this situation as soon as possible. A detailed RFO will be forthcoming once the situation is stabilized.

*** Updated at 16:00 PST on 3/13/08 ***

ServePath engineers continue working towards a resolution of the private switch network issue. Please continue to check this entry for the latest information.


*** Updated at 14:15 PST on 3/13/08 ***


ServePath and Cisco technicians have identified and isolated the private switch network issue currently affecting some ServePath customers and are currently taking steps to expeditiously resolve the issue.

This post will be updated regularly with any important information as it is obtained. A full Reason for Outage (RFO) will be provided as well.

*** END OF UPDATE ***

The ServePath Data Center facility is currently experiencing problems within the private switch network. ServePath's network engineers are working with Cisco to attempt to resolve the matter presently.

Customer Impact:

Customers with a privately addressed vlan (non-public IP addresses), Cisco ASA firewalls, Netscreen 5GT firewalls, Sonicwall firewalls, and multi-server architectures behind one public switch port may be experiencing packet loss or high latency.

COMPLETED -- Scheduled Power Maintenance: March 13, 2008 @ 08:00 PST

*** COMPLETED - 16:10 PM Pacific ***

Transfer back to the power utility is complete and ServePath/ColoServe is currently running under utility (PG&E) power. No incidents of losses of power within the data center were reported while equipment was running on diesel generators or UPS systems.

**Update @ 09:42 PST**

Power maintenance is presently under way, and the data center facility is running on generator power.

The ServePath Data Center facility will be running on generator power for up to 9 hours while Pacific Gas & Electric performs maintenance to the power supply lines on the ServePath street.

Maintenance Window:

Thursday, March 13, 2008 from 08:00 PST - 17:00 PST
(Thursday, March 13, 2008 from 16:00 GMT - 01:00 GMT)

Customer Impact:

There should be no impact to the power in the facility and no down time is expected. This notice is for advisory purposes only.

Additional Details:

The power utility PG&E will be performing maintenance on electrical service lines in the streets adjacent to ServePath's data center facility in order to accommodate the power needs of a large new building adjacent to our facility.

As with any maintenance there are potential risks, but this is an advisory notice only as we do not expect any issues. If there are any additional questions before, during, or following the maintenance, please direct them to ServePath Support Center via phone - Toll-Free 1-866-321-PATH (7284) / International +1-415-869-7000 - or open a case on-line at https://my.ServePath.com .

Tuesday, March 11, 2008

Completed -- Scheduled F5 Maintenance: March14, 2008 @ 00:00 AM PST

Maintenance Window:

Friday, March 14, 2008 from 00:00 PST - 01:00 PST
(Friday, March 13, 2008 from 08:00 GMT - 09:00 GMT)

Customer Impact:

There should be no impact to customers. This notice is for advisory purposes only.

Additional Details:

The operating system on the F5 Global Traffic Manager will be upgraded. ServePath is performing this proactive maintenance to upgrade to a more stable release of the F5 GTM.

As with any maintenance there are potential risks, but this is an advisory notice only as we do not expect any issues. If there are any additional questions before, during, or following the maintenance, please direct them to ServePath Support Center via phone - Toll-Free 1-866-321-PATH (7284) / International +1-415-869-7000 - or open a case on-line at https://my.ServePath.com .

Friday, March 7, 2008

Completed -- Scheduled Fortinet Maintenance: Mar 12, 2008 @ 9:00 PM PST

**Update @ 10:42 PM PST, 3/11/08**

The scheduled maintenance went as planned, and the Fortinet is showing expected version information.

Maintenance Window:

Tuesday, March 12th, 2008 from 9:00 p.m. - 10:00 p.m. PST
(Wednesday, March 13th, 2008 from 5:00 a.m. - 6:00 a.m. GMT)

Maintenance Details:

ServePath will be performing a proactive upgrade of the Fortinet firewall to a more stable release of the FortiOS.

Customer Impact:

During this second maintenance, your server may be unavailable for up to 5 minutes.

Contact Information:

If there are any additional questions before, during or following the maintenance, please direct them to the ServePath Support Center at 1-415-869-7000 or open a case on-line at https://my.ServePath.com .

Completed -- Scheduled Phone Maintenance: March 7, 2008 @ 6:00 PM PST

**Update @ 11:18PM PST, 3/8/08**

The scheduled maintenance was a success. All phone service has been restored.

Maintenance Window:

Date: Friday, March 7th, 2008
Time: 6:00 pm - 8:00 pm PST / 3:00 am - 6:00 am GMT on 3/8/08

Maintenance Details:

ServePath will be migrating the Phone System from one location to another. This is the second of 2 maintenance windows for this task. The first was completed successfully on Wednesday. The one listed above is for the full cut-over of the equipment (from T1/PRI circuit provisioning to the porting of numbers).

Customer Impact:

During this second maintenance, all inbound calls will be pointed to our answering services. Users with Support Issues are encouraged to log into http://my.servepath.com and open a ticket for any issues during the maintenance.

Callers with Sales concerns are advised to call the following day during normal Business Hours and will be given the option to leave a message on the Sales Voicemail box.

Callers with Billing concerns are advised to call the following day during normal Business Hours and will be given the option to leave a message on the Billing Voicemail box.

Contact Information:

As specified above, users with Support issues should log a ticket on http://my.servepath.com . Users with Billing and Sales issues should either call back the next day during normal Business Hours or leave a message in the respective voicemail boxes.

Tuesday, March 4, 2008

Completed -- Scheduled Phone Maintenance: March 5, 2008 @ 6:00 PM PST

**Update @ 10:26 PM PST, 3/5/08**

The scheduled maintenance was a success.  There will be another similar maintenance window (part 2 of 2) on Friday, March 7th, 2008 from 6:00 PM to 8:00 PM PST.

Maintenance Window:

Date: Wednesday, March 5th, 2008
Time: 6:00 pm - 8:00 pm PST / 3:00 am - 6:00 am GMT on 3/6/08

Maintenance Details:

ServePath will be migrating the Phone System from one location to another. There are currently 2 maintenance windows for this task. The one listed above is for the Pre-Test of the equipment (from T1/PRI circuit provisioning to the porting of numbers are done correctly).

Customer Impact:

During the Pre-Test period, all inbound calls shall be pointed to our answering services as well as On-Call Support cell phones. Users with Support Issues are encouraged to log into http://my.servepath.com and open a ticket for any issues during the maintenance.

Callers with Sales concerns are advised to call the following day during normal Business Hours and will be given the option to leave a message on the Sales Voicemail box.

Callers with Billing concerns are advised to call the following day during normal Business Hours and will be given the option to leave a message on the Billing Voicemail box.

Assuming everything tests positively, final migration will take place on Friday, March 7th, 2008 from 6:00 pm PST to 8:00 pm PST.

Contact Information:

As specified above, users with Support issues should log a ticket on http://my.servepath.com . Users with Billing and Sales issues should either call back the next day during normal Business Hours or leave a message in the respective voicemail boxes.