Friday, July 31, 2009

Resolved -- Level3 Network Issue

At approximately 01:10 PDT we noticed a sharp decline in traffic on our Level3 connection.  Traffic returned at approximately 01:30 PDT and we are currently investigating the cause.

Level 3 was performing a scheduled maintenance in their internal network during which they inadvertently black-holed all our inbound and outbound traffic for around 20 minutes.  Our traffic didn’t get automatically re-routed through other carriers because our BGP session with our next-hop router at Level 3 stayed up.