Hacker News Reader

Comment by jchw | Hacker News Reader

Among other things, length prefixing is annoying when streaming; it basically requires you to buffer the entire message, even if you could more efficiently stream it in chunks, because you need to know the length ahead of time, which you may very well not.

View on HN

Remember FLV? What about MP4? Surprise! Both use length prefixing, and stream perfectly fine.

Length-prefixing is not a problem for streaming. Hierarchical data is, but even then, you have stuff like SAX (for XML).

The problem with Protobuf and why you cannot stream it is that it allows repetition of the same field, and only the last one counts. So, length-prefixing or not, you cannot stream it, unless you are sure that you don't send any hierarchical data (eg. you are sending a list of integers / floats).

Ah, also, another problem: "default fields". Until you parsed the entire message you don't know if default fields are present and whether they need to be initialized on the receiving end.

> length prefixing is annoying when streaming

This can be avoided by magic number. If length is 0, then message length isn't known.

That does leave one problem: you still need a way to segment your stream. Most length-prefixed framing schemes do not have any way to segment the stream other than the length prefix. What you wind up wanting is something like chunked encoding.

(Also, using zero as a sentinel is not necessarily a good idea, since it makes zero length messages more difficult. I'd go with -1 or ~0 instead.)

Using `-1` would require using a singed integer for the length, which I guess could be done if you're fine with having the maximum length be half as long, but that also raises the question of what to do with the remaining negative values; what does a length of -10 mean?

I thought -0 is only something in floating point numbers, not integers, and using floats for the length of a message sounds like a nightmare to me.

Ah, I was a little unclear. I mean ~0 as in NOT 0, an integer with all bits set. This is also the same as -1 in two's compliment. So basically, I'm suggesting you use the maximum unsigned integer value as a sentinel. That doesn't work if you're using a variable-length unsigned integer like base128vlq, but if you're doing base128vlq you could always make a special sentinel for that (e.g. unnecessarily set the high bit and follow it with a zero byte; this would never normally appear with a base128vlq.)

Or it doesn't have a length. For messaging protocols - or in general - magic should be avoided at all times.

If you have random access, you could leave some space and then go back and fill in the actual length value. Would work better with fixed size integer as you know ahead of time how much space to leave.

If you’re streaming, you generally don’t have random access.

Somehow I missed the ‘streaming’ part… my bad

Just leave some empty electromagnetic waves in the cables before your message, got it.

Reply on news.ycombinator.com