Notes from their massive DWDM system outage earlier on the 27th, which preceded the 911 outage:
2018-12-28 13:35:00 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT remains.
2018-12-28 13:27:30 GMT - The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes. CenturyLink NOC management remains in contact with the equipment vendor to obtain updates as restoration efforts continue.
2018-12-28 11:04:24 GMT - CenturyLink continues to work with the Equipment Vendor to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Client services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes.
2018-12-28 10:05:18 GMT - CenturyLink NOC Management reports steady progression of node recovery and restoral of client services. In addition to the remote node recovery process, Field Operations continue to dispatch and assist the Equipment Vendor with local equipment login.
2018-12-28 08:51:29 GMT - CenturyLink NOC Management has advised that repair efforts are steadily progressing, and services are incrementally restoring. The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels at this time. There have been additional restoration steps identified for certain nodes, which includes either line card resets or Field Operations dispatches for local equipment login, that have impeded the restoration process. Various repair teams are working in tandem on these actions to ensure that services are restored in the most expeditious method available. Restoration efforts are ongoing.
2018-12-28 07:12:32 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. Additional information on repair progress will be available from the Equipment Vendor by 07:30 GMT. Information will be relayed as soon as it is obtained.
2018-12-28 06:00:01 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT remains.
2018-12-28 04:58:44 GMT - CenturyLink engineers in conjunction with the Equipment Vendor¿s Tier IV Technical Support team have identified the elements causing the impact to customer services. Through the filters being applied and the removal of the secondary communication channels, it is anticipated services will be fully restored within four hours.¿We apologize for any inconvenience this caused our customers. Additional details regarding details of the underlying cause will be relayed as available.
2018-12-28 04:09:31 GMT - The Equipment Vendor¿s Tier IV Technical Support team in conjunction with CenturyLink Tier III Technical Support continues to remotely work to remove the secondary communication channel tunnels across the network until full visibility can be restored, as well as applying the necessary polling filter to each of the reachable nodes.
2018-12-28 02:53:38 GMT - The Transport NOC has confirmed that cooperative efforts remain ongoing to remove the secondary communication channel tunnel across the network until full visibility can be restored, as well as applying the necessary filter to each of the reachable nodes. It has been confirmed that both of these actions are being performed remotely, but an estimated time to complete the activities is not available at this time.
2018-12-28 01:58:56 GMT - Once the card was removed in Denver, CO it was confirmed that there was no significant improvement. Additional packet captures, and logs will be pulled from the device with the card removed to further isolate the root cause. The Equipment vendor continues to work with CenturyLink Field Operations at multiple sites to remove the secondary communication channel tunnel across the network until full visibility can be restored. The equipment vendor has identified a number of additional nodes that visibility has been restored to, and their engineers are currently working to apply the necessary filter to each of the reachable nodes.
2018-12-28 00:59:04 GMT - Following the review of the logs and packet captures, the Equipment Vendor's Tier IV Support team has identified a suspected card issue in Denver, CO. Field Operations has arrived on site and are working in cooperation with the Equipment Vendor to remove the card.
2018-12-27 23:57:16 GMT - The Equipment Vendor is currently reviewing the logs and packet captures from devices that have been completed, while logs and packet captures continue to be pulled from additional devices. The necessary teams continue to remove a secondary communication channel tunnel across the network until visibility can be restored. All technical teams continue to diligently work to review the information obtained in an effort to isolate the root cause.
2018-12-27 22:52:43 GMT - Multiple teams continue work to pull additional logs and packet captures on devices that have had visibility restored, which will be scrutinized during root cause analysis. The Tier IV Equipment Vendor Technical Support team in conjunction with Field Operations are working to remove a secondary communication channel tunnel across the network until visibility can be restored. The Equipment Vendor Support team has dispatched their Field Operations team to the site in Chicago, IL and has been obtaining data directly from the equipment.
2018-12-27 21:35:55 GMT - It has been advised that visibility has been restored to both the Chicago, IL and Atlanta, GA sites. Engineering and Tier IV Equipment Vendor Technical Support are currently working to obtain additional logs from devices across multiple sites including Chicago and Atlanta to further isolate the root cause.
2018-12-27 21:01:26 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC was engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO and New Orleans, LA a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. Tier IV Equipment Vendor Technical Support continues to review equipment logs from the sites where visibility was previously restored. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.