Hacker News Reader

Comment by nostrademons | Hacker News Reader

Most of the time you could set reasonable bounds on what you expect the length to be. How often does your custom protocol require multi-gigabyte messages?

If you can't, sanity check against negative numbers, make sure you check the return value of malloc(), and set a timeout on actually reading that much data. If the malloc fails, close the connection. If the timeout fails, close the connection and free the resources. On the public Internet, you probably don't want to send an error message, since it's just exposing internal system information an attacker could exploit. (In debug mode, running internally, you probably do want to expose this.)

View on HN

This sounds like it will lead to annoying arbitrary limits.

You'll be just as angry when you find out some application only lets you send it messages up to 1 MB. What a stupid restriction, you'll think!

Nah. I don't think it's stupid at all that email is generally limited to 10MB, for example.

Which is more annoying? Being unable to send your 1G message, or the service being down because it's being DOSed by 1G messages?

A DOS attack could be performed using any length of message, although the effort required to kill the server by making it allocate all it's memory would be much lower.

Set limit = 10M, log request sizes, when you start seeing requests with size >1M, consider raising limit.

If you use 4 bytes for size, which is fairly convenient, you get 16 megabytes before any 1s show up in the most significant byte.

If you expect that it might be an accidental ASCII character, the smallest one you are likely to accidentally receive is space. 32 x 16MB = 512MB.

If that's still not a large enough atomic message to satisfy, allocate 8 bytes for the size. If sneaky ASCII creeps in to the most significant byte, it's going to at least be 2^57, or 512 exabytes.

Plus, this is just the first layer of defense. It's an easy and cheap one that keeps you from overallocating, but it's not the end of the story. You still need to examine the rest of the message for validity, and throw it out as soon as possible if you find it's invalid.

And, finally, if you are taking the "allocate a buffer to hold the message" approach, and you might be receiving multiple messages at a time, you have to consider the possibility of receiving multiple spurious messages at once. The only thing worse than a process spuriously allocating a gigabyte and trying to parse stuff into it is a process spuriously trying to do that a few thousand times per second.

I can just imagine a HN article 'Check out the crazy reason that this protocol is limited to 512MB!'

There's barely any difference between a limit of 512MB vs. a limit of 2GB/4GB. Almost nobody is going to split that hair. You either design for reasonably-small messages, or you design for 'unlimited' size and go with a 64 bit size field. Either way shaving a few bits off the top is harmless.

But you shouldn't be allocating 100MB+ upfront anyway. No matter how big of a message you allow.

This reminds me of the other article on HN recently. Using calloc makes it cheap because it's using the kernel's CoW. Of course you probably want to use that CoW functionality explicitly in this case.

Yeah, everybody goes "640K OUGHT TO BE ENOUGH FOR ANYBODY, LOL." But if you're designing (say) a custom in-house protocol for transmitting telemetry, then "what happens in 20 years when someone tries use it for bulk data transfer for some reason" really shouldn't be higher on your list of priorities than "what happens when it receives literally any data besides what it was expecting." Hypothetical future expandability is good to consider, but not as important as present-day intended use.

Reply on news.ycombinator.com