On August 30th, the U.S. Internet service provider CenturyLink suffered a major technical failure. A misconfiguration in one of its data centers caused damage to the entire Internet network.

Involving both a firewall and BGP routing, this technical failure spread outside CenturyLink's network. This error also affected other Internet Service Providers (ISPs), ultimately causing connectivity problems for many other companies.

 

The technology giants whose services were disrupted include Amazon, Twitter, Microsoft (Xbox Live), EA, Blizzard, Steam, Discord, Reddit, Hulu, Duo Security, Imperva, NameCheap, OpenDNS, and many others.

Also severely affected, Cloudflare said that CenturyLink's outbound spread problem has resulted in a 3.5 percent drop in global Internet traffic. This would make it one of the largest Internet outages ever recorded.

 

According to CenturyLink's status page, the problem originated at its data center in Mississauga, a city near Toronto, Canada.

The ISP says the root cause of the incident is an incorrect Flowspec ad. Its Mississauga data center reportedly sent an incorrect Flowspec announcement, preventing the company's BGP routes from taking root.

As CenturyLink's incorrect Flowspec command caused some of the routers in its network to fail, some of these routers also began announcing incorrect BGP routes to other nearby "tier 1" Internet services. This caused other networks to fall into a domino effect.

 

CenturyLink solved the problem by taking the unprecedented step of telling all other "Tier 1" ISPs to disconnect and ignore all traffic coming from its network. Companies rarely make these kinds of decisions because it results in a total loss of connectivity for all their customers.

As a result, the ISP had to reset all equipment and start with clean BGP routing tables, a process that took nearly seven hours, the company said.

 

 

 Read the article

 

Source : ZDNet