Level 3 blames huge network outage on human error

Level 3 Communications has cited a “configuration error” as the root cause of its nationwide network outage on Tuesday.

Here’s the public statement issued by the Broomfield, Colo., service provider:

On October 4, our voice network experienced a service disruption affecting some of our customers in North America due to a configuration error. We know how important these services are to our customers. As an organization, we’re putting processes in place to prevent issues like this from recurring in the future. We were able to restore all services by 9:31 a.m. Mountain time.

(UPDATED on Oct. 14, 2016) Level 3 got more specific with customers, issuing a Reason for Outage (RFO) Summary (shared by a Network World reader) headlined “Repair Area: Human Error Occurrence” and that read in part: “Investigations revealed that an improper entry was made to a call routing table during provisioning work being performed on the Level 3 network. This was the configuration change that led to the outage. The entry did not specify a telephone number to limit the configuration change to, resulting in non-subscriber country code +1 calls to be released while the entry remained present. The configuration adjustments deleted this entry to resolve the outage.”

Social media sites such as Reddit and Twitter erupted on Tuesday morning with inquiries and complaints about the outage from Level 3 customers, as well as customers of other big carriers like AT&T and Verizon that were affected by the outage. Speculation for the outage ranged from possible fiber cuts to more outlandish theories.

Leave a Reply

Your email address will not be published. Required fields are marked *