Out of interest, did you use skiplists to reassemble the streams in order?
If you've already asked, one cool part of that technology is that the order of received packets is not important to assemble the stream. Basically every 1 second of video is reassembled without importance to the order of packets received. You need N packets to assemble a "data frame", IIRC pending incomplete data frames were stored in a simple hash table, but honestly it was so long ago I don't remember.