Comment by MichaelGG | Hacker News Reader

Comment by MichaelGG | Hacker News Reader

Had a VoIP network with TB+ of SIP packets each day. Customers demand resolution to problems from days ago, so having a time-travelling, content-aware PCAP was necessary. At 50K packets/sec, piping tshark to mysql and 11 indexes simply wouldn't cut it. We spent $$$$$$ on a commercial system that didn't work so well.

I slowly reinvented the basics of an information retrieval system (the curse of not having taken CS classes). Came up with the idea of a log-structured merge tree, made easier by this being a write-once database. Got some inspiration from the original Google paper. But most of it was just figuring out the least number of actions needed to retrieve info.

I published the core DB part, which maps an int64 (index hash value) to an int64 (docId) and stores in an efficient format (on our data, ~2.3 bits per packet). http://github.com/michaelgg/cidb - I couldn't find an existing library that has zero/low per-record overhead.

On a Q6600 and a single 7200RPM platter, I was able to index a TB of SIP a day and provide fairly quick flow reconstructions going back as far as disk space allowed. On a quad-core i7 parsing+indexing was over 1Gbps.

Company impact was huge, because we could suddenly troubleshoot things in minutes instead of hours. A few years later, after I was gone, I heard they were still using it. Neat.

This was all in F#, which presented fun challenges regarding optimization. Lots of unsafe code and manual memory management. SSE would improve varint encoding - the CLR generated code is a joke in comparison.

Last month, I dropped this lib in as a replacement for storing routing information in a telecom app and dropped RAM requirements from 6GB to 1GB.

On the downside, I'm sure any compsci student could build a similar thing in a week, and probably they do so for school projects. But to a lot of app-level developers, this kind of algorithmic work is sorta magic for some reason.

Sounds like a great tool -- I would certainly use something like this during my telecom days. At the end, did you still use tshark to pipe input data to your database?

No, I wrote a fully custom SIP parser and database format. If I was doing it again, I'd look at stealing a SIP parser from another project. Scale out and don't worry about every CPU cycle.

I thought about commercializing it but interest seemed lukewarm.

But how did you capture the actual bytes on the network (that you would later parse as SIP)?

libpcap. For my first pass I just setup a mirror port (SPAN) port from a core switch. libpcap has efficient filters so non-SIP (non port 5060) traffic was just dropped.

Looking forward: Intel 10G NICs have little processor in them, and there's a library called DNA (Direct NIC Access) that bypasses kernel mode for packet capture/transmit. Makes it super efficient and possible to do line-rate 10G apps on commodity hardware.

Those processors can also distribute packets to specific cores, which helps solves the next problem of scaling the capture part beyond a single core. Looking beyond 10G to 40G and beyond, I'd imagine something similar to Etherchannelling would be a cheap way to leverage existing hardware into breaking up the load. Unfortunately a low-level "dumb" load balancing system will split up dialogs so your indexing and compression gets a bit less efficient.

One of my semi-open research projects is to do line-rate SIP DDoS protection. So far, no one (not commercially, not academically) has any magic bullet and it would appear that almost every SIP network out there is trivially DoS-able.

State-of-the-art is along the lines of "we push a whitelist into our switches" (which, actually, technically doesn't even work with SIP since multiple IPs can get involved with the signalling, IPs unrelated to original hosts - although I've only seen this once or twice in production).

As far as I can tell, it's mainly an engineering problem - just do the grunt work of writing smart code. But there's no market for it until VoIP providers start suffering real serious pain - until then, I don't think telecom cares about security, DoS or otherwise.

I'm a senior compsci student at one of the largest (by # of students) universities in the nation. I assure you, most of my classmates could not do such a thing in a week!

Reply on news.ycombinator.com