Hacker News Reader

Comment by aaron42net | Hacker News Reader

Skimming the bucket code, it appears that the rate-limiting information is stored in a map. It doesn't appear to implement an LRU or attempt to clean up stale entries.

If I feed this user-supplied keys such as IP address or cookies, doesn't this mean it will grow without bound?

View on HN

Golimit tries to reuse the allocated keys counter entry in case the counter has expired and its not collected by GC. There is a very simple GC logic where GC is invoked periodically, and scans for all buckets one by one and cleans them up. The GC invocation frequency can be configured by setting "gcinterval" property in yaml.

If it is rate limiting IPv4 address space, then you only need a few gigs to store them all... perhaps this is why there is no garbage collection.

Agree with this. Space used by keys can be reduced by hashing the key. Or for IPs they can be converted to an int value.

an ip as an int will be converted back to binary to be stored, so will still require 4 bytes to store (uncompressed). unless i'm missing something?

Yes that's what I meant. The best size from Ipv4 is 4 bytes. Its better instead of using 111.111.111.111

In addition to the 15 bytes used by the string itself, which will be rounded up to a minimum of 16 bytes in RAM (and probably larger), the string carries additional words for pointers and data, whereas the int is just 4 bytes and done. Go can use it as a value type so there doesn't even have to be another couple of words wrapping it as an "object" or something. Plus you can potentially start optimizing for the int case with various clever things more easily, because it's so tight it gets down into the realm where bit bashing games can be used to potentially avoid cache line hits, etc. whereas the string representation isn't so amenable to that.

gotcha, it didn't immediately occur to me to store it as a string, so that's why i was puzzled.

Reply on news.ycombinator.com