Comment by carlsborg | Hacker News Reader

Comment by carlsborg | Hacker News Reader

How does a single network card emitting bad packets effect other sites?

> investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source packet did originate from this card.

> Support shifted focus to the application of strategic polling filters along with the continued efforts to remove the secondary communication channels between select nodes.

And then

> By 2:30 GMT on December 29, it was confirmed that the impacted IP, Voice, and Ethernet Access services were once again operational. Point-to-point Transport Waves as well as Ethernet Private Lines were still experiencing issues as multiple Optical Carrier Groups (OCG) were still out of service.

And finally

> The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.

Looks like the root cause analysis has a way to go. Addendum says:

> The CenturyLink network continued to rebroadcast the invalid packets through the redundant (secondary) communication routes.. These invalid frame packets did not have a source, destination, or expiration and were cleared out of the network via the application of the polling filters and removal of the secondary communication paths between specific nodes. The management card has been sent to the equipment vendor where extensive forensic analysis will occur regarding the underlying cause, how the packets were introduced in this particular manner. The card has not been replaced and will not be until the vendor review is supplied. There is no increased network risk with leaving it unseated. At this time, there is no indication that there was maintenance work on the card, software, or adjacent equipment. The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.

It will be almost impossible to get a good idea of the actual RCA from public reports, since even if they are accurate, we would need a complete understanding of the topology of the network in question to understand them. That said, the public descriptions will be filtered through so many layers of PR and lawyers as to be incomprehensible. "invalid frame packet"? Were they invalid frames or invalid packets? As fun as it is to gawk, I don't think we'll ever get a good picture of what happened and what CenturyLink did wrong, if anything, to allow such an outage to take place.

My reading of the tea leaves is that the invalid packets where for some low level control protocol.

I’ve had invalid spanning tree (protocol typically used to prevent loops in networks) packets cause a trunk link to flap as the invalid packet made the switch think its only trunk link (the link to the rest of the network) was part of a loop and shut it down. When the link went down, it could no longer get the bad packets, so after a delay it would enable the trunk again and get the invalid packets again.

I've seen plenty of networks where there is a packet storm, then STP disables the link and fixes it, then 30 seconds later STP re-enables the link and the packet storm resumes...

I have a very low opinion of STP. I'm also not a fan of the fad to have very flat networks where there are very few routers and instead everything is switches in one gigantic subnet. Packet storms are notoriously difficult to track down on big networks.

>I have a very low opinion of STP.

That's not a very radical opinion in network engineering circles. No one ever liked it due to it's non forwarding links for loop prevention but it was good enough to work until we discovered it's successor.

> until we discovered it's successor

Which is?

Ensuring all layer 2 domains are loop free by design and moving all redundancy to layer 3 and routing protocols.

> Ensuring all layer 2 domains are loop free by design

Impossible, unfortunately, or we wouldn't need STP.

> and moving all redundancy to layer 3 and routing protocols.

Not always possible either, sadly. Works well for new deployments, can difficult to retrofit into older deployments depending on the scale and applications involved.

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server:

    lookupByCity(...) {
        ....
        try {
            conn = connectionPool.getConnection();
            stmt = conn.createStatement();
            ...
         } finally {
             if (stmt != null) {
                 stmt.close();
             }
             if (conn != null) {
                 conn.close();
             }
         }
    }

close() can throw, and in the circumstance of the outage it did for stmt, leading to the connection not getting closed and eventually the pool being exhausted with every thread blocked waiting for a connection. It's an interesting chain of failures, arguably the presence of such a chain is the real root cause, rather than the unhandled sql exception.

"Release It" is an awesome book. I contributed a couple FindBug Java checkers to warn about some of the problems the book describes.

The case you mention above might have been prevented by using checked Java exceptions. Our programming languages and tools could be doing a lot more to catch these problems at compile time or make them impossible by language or API design.

I think that was taken into account when the "try with resources" statements were added to Java.

As you said it would be nice if the languages could make these situations impossible.

I think its a bit of a chicken and egg problem because some language issues don't come to light until people are using it and it is too late.

I know people like to hate on errors as values as in Go, but I think exceptions are worse when it comes to unexpected side effects and this is a prime example!

Yes; although in GC’ed languages like Go and JS it’s still very easy to leak OS level resources like file handles because you still need to remember to close() them. (Although go’s defer blocks are a fantastic assist here).

This is one area Rust really excels - the same mechanism for making sure memory gets cleaned up also automatically closes network sockets and file descriptors when they go out of scope. Even in the case of errors it’s impossible to forget to clean up. That entire finally block is unnecessary in rust.

You're talking about the Drop trait?

I'm learning Rust coming from Go. It looks cool, but it also concerns me how most data structures in the stdlib use unsafe blocks to defeat the borrow checker. This is not the point of Rust, I would have thought?!

One of the criteria for belonging in the standard library is “needs a lot of unsafe to implement”, so the standard library has more unsafe code than your average codebase.

Beyond that, to some degree, it is the point of Rust: limit unsafe things so that you can reason about them more easily. The CPU is inherently not safe, so it has to exist on some level. rust gives you tools to manage this.

My favorite is constructors of things that taken input/output streams that themselves can throw. It becomes very verbose to ensure that the unwinding happens correctly.

the presence of such a chain is the real root cause, rather than the unhandled sql exception.

This is really interesting and something which bugs me about root cause analysis and it's a neat coincidence that this has been quoted relative to an aviation incident.

In aviation, incidents and accidents are investigated with the understanding that there is never a single cause of an accident. It's known as the swiss cheese model. All the holes in the swiss cheese have to line up for something to go wrong. Even in a seemingly simple "pilot error" accident, there are years of initial and recurrent training factors, ergonomic and human factors and so on which all lead to the event. It's exceedingly rare for a single "root cause" to be the whole story.

Medicine is starting to adopt techniques learned from aviation like checklists, crew resource management and no-blame, swiss-cheese accident investigations. I am hopeful that the software industry will take similar lessons over the next decade or so.

> like checklists, crew resource management and no-blame, swiss-cheese accident investigations. I am hopeful that the software industry will take similar lessons over the next decade or so.

The software industry that programs space craft?

The software industry is vast and not every system involves copious amounts of human decision making. Often the idea of the root system cause, and the root process cause(software construction, operations, etc) cause is separable.

I would say that aviation is almost inverted in the that regard compared to booking systems, banking systems, and most of what software engineers are exposed to. A person can not fly from Dallas to Chicago without many, many human decisions being involved. However, a packet traveling from Dallas to Chicago involves nearly zero new human interactions.

Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.

Yep. It sounds like the packets were coming from this nic and the backbone routers had no logic to filter out those kinds of invalid packets. No SRC/DST/TTL should be a big red flag, but it seems like they didn't have a rule for that.

Talk about a needle in a hay stack.

Traditional IP filters are typically useless against packet storms because all of the packets are technically valid, there are just way too many of them.

I used to look after an office location that had three racks with older generation HP Procurve switches. About once every 4-6 months, the whole lot would go haywire and the only way to settle down the network was to switch them all off, wait, and then power up again. We never did track down the root cause and could never make it happen on demand,

Could be a totally valid solution ... for an equipment rental place. Pretty sure people died that didn't have to with this 911 outage.

Oh god, I had to deal with some procurve switches in a past life. They were a nightmare to manage and unreliable.

We actually ended up replacing them on 2 racks with 24port unmanaged gbit 3com ones we bought from some high street electronics shop one night during an outage... Not a fun memory..

(Of course, those were later replaced with some proper equipment)

Every time it comes to “root cause analysis” (quoting makes sense in this context) I think of https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...

The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s.

There are lots of proximate reasons that a failure can avoid all of your sanity checks, but the simplest is when your organization is a little insane. I try to steer the conversation toward avoiding surprises, other times toward ergonomics (don’t rely so much on humans in the moment to do the right thing).

Often people don’t fight me too much on that, but sometimes it’s a near thing. Lots of senior devs are senior because they keep everybody else down, so blaming human error for an outage seems perfectly reasonable to them.

> The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s.

Agreed. I think it’s a case for running a Joy Driven Development operation first, and looking to organizational responsibility first (versus looking for somewhere to cast blame), but practically, that probably too utopian to bear out in practice. No reason not to keep it in mind, though.

One example might be "ip helper" setups in a Cisco router. People use it to forward DHCP requests over a WAN. One broken SIP device, no flood control, and...bam, your WAN is hosed.

Ip helper usually only shows up in the context of DHCP so its easy to forget it forwards a lot more than just DHCP broadcasts.

Reply on news.ycombinator.com